Methods, systems, apparatuses, and techniques for employing augmented reality and virtual reality

ABSTRACT

Techniques for employing augmented reality or virtual reality information are presented. An information management component (IMC) of an augmented reality device (ARD) can monitor and detect user activities and conditions in area in proximity to ARD. Based on user activities and conditions, IMC can determine augmented reality information that can enhance user experience, performance of user activities, or security and safety of user. IMC can present, via an interface component of ARD, the augmented reality information to the user. The augmented reality information can relate to user location; navigation by the user; tasks to be performed by the user; product assembly; maintenance work; system or product design or configuration; remote control of assembly, maintenance, design, or configuration; environmental and/or hazardous conditions; security, identification, and authentication of users; or training the user to perform tasks. IMC can translate information from a language to a different language of the user.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims is a continuation of, and claims priority toeach of, U.S. patent application Ser. No. 15/807,247 (now U.S. Pat. No.10,866,631), filed Nov. 8, 2017, and entitled “METHODS, SYSTEMS,APPARATUSES, AND TECHNIQUES FOR EMPLOYING AUGMENTED REALITY AND VIRTUALREALITY,” which claims priority to U.S. Provisional Patent ApplicationNo. 62/419,499, filed Nov. 9, 2016, and entitled “VIRTUALREALITY/AUGMENTED REALITY DEVICE,” the entireties of which applicationsare hereby incorporated by reference herein.

TECHNICAL FIELD

The subject application relates generally to methods, systems,apparatuses, and techniques for employing augmented reality and virtualreality.

BACKGROUND

The subject matter disclosed herein relates to virtual reality andaugmented reality. Virtual reality is a computer-related technology thatcan use a device, such as a headset device, to simulate the presence ofa user in a virtual multi-dimensional environment. Augmented reality canprovide a view of a real multi-dimensional environment, wherein the viewcan be supplemented with other information regarding the realenvironment to augment the perception of the real environment by theuser.

The above-described description is merely intended to provide acontextual overview relating to augmented reality and virtual reality,and is not intended to be exhaustive.

BRIEF DESCRIPTION

The following presents a simplified description in order to provide abasic understanding of some aspects described herein. This descriptionis not an extensive overview nor is intended to identify key/criticalelements or to delineate the scope of the various aspects describedherein. Its sole purpose is to present some concepts in a simplifiedform as a prelude to the more detailed description that is presentedlater.

Disclosed is a device comprising a set of sensors configured to detectconditions associated with at least one of a user or an environmentassociated with the user, wherein the set of sensors generate sensordata in response to detection of the conditions. The system alsocomprises a memory that stores machine-executable components, aprocessor, operatively coupled to the memory, that executesmachine-executable components. The machine-executable components cancomprise an information management component configured to analyze thesensor data, determine a context associated with the user based on aresult of the analysis of the sensor data, and determine customizedinformation based on the context of the user, wherein the contextrelates to at least a location of the user. The system further cancomprise an interface component configured to present the customizedinformation to the user to facilitate performance of at least one task.

Also disclosed is a method comprising sensing, by a system comprising aprocessor, conditions associated with at least one of a user or anenvironment associated with the user, wherein sensor information isgenerated based on the sensing. The method also can include determining,by the system, a context associated with the user based on a result ofanalyzing the sensor information. The method further can comprisedetermining, by the system, customized data based on the context of theuser, wherein the context relates to at least a location of the user inan area associated with the environment. The method also can comprisepresenting, by the system, the customized information to the user tofacilitate performance of at least one task.

The disclosed subject matter further comprises a system that can includea group of sensors configured to sense conditions associated with atleast one of a user or an environment, wherein at least one sensor ofthe group of sensors generates sensor information in response to thesensing of the conditions. The device also can comprise a memory thatstores machine-executable components, and a processor, operativelycoupled to the memory, that executes machine-executable components. Themachine-executable components can include an information managementcomponent configured to analyze the sensor information, determine acontext associated with the user based on a result of the analysis ofthe sensor data, and generate customized data based on the context ofthe user, wherein the context relates to at least a location of the userwith respect to the environment. The device further can comprise aninterface component configured to present the customized data to theuser to facilitate performance of at least one task.

To the accomplishment of the foregoing and related ends, certainillustrative aspects are described herein in connection with thefollowing description and the annexed drawings. These aspects areindicative of various ways which can be practiced, all of which areintended to be covered herein. Other advantages and beneficial featuresmay become apparent from the following detailed description whenconsidered in conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an example system that can employan augmented reality device (ARD) to facilitate performance and/orenhancement of various activities and tasks, in accordance with variousembodiments and aspects of the disclosed subject matter.

FIG. 2 depicts a block diagram of an example sensor component that cancomprise a set of sensors that can be used in conjunction with the ARD,in accordance with various embodiments and aspects of the disclosedsubject matter.

FIG. 3 illustrates a block diagram of an example interface componentthat can comprise a set of interfaces that can be used in conjunctionwith the ARD, in accordance with various embodiments and aspects of thedisclosed subject matter.

FIG. 4 depicts a diagram of various example visual presentations in thefield of view of a user in connection with an assembly process, inaccordance with various aspects and embodiments of the disclosed subjectmatter.

FIG. 5 illustrates a diagram of various example visual presentations inthe field of view of a user in connection with performing maintenancetasks or other tasks, in accordance with various aspects and embodimentsof the disclosed subject matter.

FIG. 6 presents a diagram of an example visual presentation thatcomprises a virtual display that includes respective risk levels ofrespective components in the field of view of the user through the ARD,in accordance with various aspects and embodiments of the disclosedsubject matter.

FIG. 7 depicts a diagram of various example visual presentationscomprising data and/or visual indicators relating to components of asystem that are in the field of view of (or otherwise in proximity to)the user through the ARD, in accordance with various aspects andembodiments of the disclosed subject matter.

FIG. 8 illustrates a diagram of various example visual presentationsthat can facilitate enabling a user to identify, focus on, and/or selectdesired items of data associated with a system, in accordance withvarious aspects and embodiments of the disclosed subject matter.

FIG. 9 illustrates a diagram of example visual presentations that can bepresented to the user of the ARD to enable the user to safely navigatethrough an area and around objects, barrier, and/or hazardous conditionsin the area, in accordance with various aspects and embodiments of thedisclosed subject matter.

FIG. 10 illustrates a block diagram of an example system that can employan ARD in a communication network to facilitate performance and/orenhancement of various activities and tasks, in accordance with variousembodiments and aspects of the disclosed subject matter.

FIG. 11 depicts a block diagram of an example information managementcomponent, in accordance with various embodiments and aspects of thedisclosed subject matter.

FIG. 12 illustrates a flow chart of an example method that can determineand present customized information, comprising augmented realityinformation, to a user to facilitate performance of tasks by the user,in accordance with various aspects and embodiments of the disclosedsubject matter.

FIG. 13 presents a flow chart of an example method that can determineand present customized information, comprising augmented realityinformation, to facilitate generating a map comprising a travel routethat includes indicators for hazards and/or obstacles, in accordancewith various aspects and embodiments of the disclosed subject matter.

FIG. 14 depicts a flow chart of an example method that can determine andpresent customized information, comprising augmented realityinformation, to facilitate performance of product assembly tasks, inaccordance with various aspects and embodiments of the disclosed subjectmatter.

FIG. 15 illustrates a flow chart of an example method that can determinehazards in an area, and can determine and present customizedinformation, comprising augmented reality information, via an ARD tofacilitate notifying or advising an ARD user of the hazards in the area,in accordance with various aspects and embodiments of the disclosedsubject matter.

FIG. 16 is an example computing and/or operating environment.

FIG. 17 is an example computing and/or networking environment.

DETAILED DESCRIPTION

The subject disclosure is now described with reference to the drawings,wherein like reference numerals are used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding thereof. It may be evident, however, that the disclosedsubject matter can be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate a description thereof.

Various aspects or features will be presented in terms of systems thatmay include a number of devices, components, modules, and the like. Itis to be understood and appreciated that the various systems may includeadditional devices, components, modules, etc. and/or may not include allof the devices, components, modules etc. discussed in connection withthe figures. A combination of these approaches also can be used.

The subject matter disclosed herein relates to virtual reality andaugmented reality. Virtual reality is a computer-related technology thatcan use a device, such as a headset device, to simulate the presence ofa user in a virtual multi-dimensional environment. Augmented reality canprovide a view of a real multi-dimensional environment, wherein the viewcan be supplemented with other information regarding the realenvironment to augment the perception of the real environment by theuser.

In various types of situations, product assembly, maintenance work,industrial work (e.g., industrial automation system work), design andconfiguration of systems (e.g., industrial automation systems) orproducts, general life situations, or other situations, various issuescan arise. For example, a user can be doing assembly work on a product,wherein various issues can arise, such as issues regarding havingdesired (e.g., suitable) information at a desired time to facilitate theperformance of an assembly task, the use of an incorrect part during theassembly, the existence of a hazard associated with assembly of theproduct, and/or lack of knowledge or training of the user with respectto performing tasks associated with assembly of the product. As anotherexample, there can be issues with regard to performing maintenance workor industrial work, such as, issues regarding having desired (e.g.,suitable) information at a desired time to facilitate the performance ofa maintenance task, the existence of a hazard in an area where themaintenance or industrial work is to be performed and/or with regard tothe performing of the maintenance task or industrial work task, othersafety-related issues (e.g., issue as to whether the user or anotherperson has the experience, skills, and/or training to perform the task),lack of knowledge or training of the user with respect to performingtasks associated with maintenance or industrial work, and/orsecurity-related issues (e.g., issue as to whether the user or anotherperson is authorized to be in the area they are at and/or are theyauthorized to perform the task). As a further example, there can beissues with regard to designing or configuration of systems or products,such as issues regarding having desired (e.g., suitable) information ata desired time to facilitate the design and configuration of the systemor product, issues regarding how to overcome certain design orconfiguration obstacles in connection with the design or configurationof the system or product, lack of knowledge or training of the user withrespect to performing tasks associated with the design or configurationof the system or product, safety-related issues (e.g., issue as towhether the user or another person has the experience, skills, and/ortraining to perform tasks related to the design or configuration of thesystem or product), security-related issues (e.g., issue as to whetherthe user or another person is authorized to be in the area they are atand/or are they authorized to perform tasks related to the design orconfiguration of the system or product)

The disclosed subject matter can overcome these and other issues toenhance the user experience of the user, enhance the activities and/orproductivity of the user, enhance the assembly of products, improve theperformance of maintenance work, enhance the designing and configurationof systems or products, provide assistance and/or or training to theuser regarding various tasks, enhance the life of the user with regardto general life situations, and/or enhance the safety and security ofthe user and/or other people, etc.

To that end, techniques for employing augmented reality or virtualreality information to a user are presented. An information managementcomponent (IMC) of an augmented reality device (ARD) can monitor anddetect user activities and conditions in area in proximity to the ARD.Based on user activities and conditions, the IMC can determine augmentedreality information that can enhance user experience, performance ofuser activities, or security and safety of user. The IMC can present,via an interface component of the ARD, the augmented reality informationto the user. The augmented reality information can relate to, forexample, user location; navigation by the user; tasks to be performed bythe user; product assembly; maintenance work; system or product designor configuration; remote control of assembly, maintenance, design, orconfiguration; environmental and/or hazardous conditions; security,identification, and authentication of users; or training the user toperform tasks.

The IMC also can translate information from a language to a differentlanguage of the user and can facilitate presenting the translatedlanguage to the user via the interface component of the ARD. The IMCfurther can provide information (e.g., augmented reality information)regarding locales, people, etc., to the ARD user, and/or provideinstruction regarding various subjects, can provide entertainment (e.g.,music, movies, electronic games, . . . ) to the user, can notify theuser when the user is impaired (e.g., sleepy, inebriated, . . . ), canprovide reminders of events, appointments, or tasks, and/or can enableconnection (e.g., communicative connection) with other people (e.g.,family, friends, . . . ) or businesses.

These and other aspects and embodiments of the disclosed subject matterwill now be described with respect to the drawings.

FIG. 1 illustrates a block diagram of an example system 100 that canemploy an augmented reality device (ARD) to facilitate performanceand/or enhancement of various activities and tasks, in accordance withvarious embodiments and aspects of the disclosed subject matter. Thesystem 100 can comprise the ARD 102. While the ARD 102 is referenced asan augmented reality device, it is to be understood and appreciated thatthe ARD 102 also can employ virtual reality functions and features, inaddition to augmented reality functions and features, as more fullydescribed herein.

The ARD 102 can be, for example, an augmented reality, virtual realityheadset that can be worn by a user. The ARD 102 can comprise and canutilize headset hardware, firmware, and software routines to facilitateperforming various operations and tasks, such as disclosed herein. Theheadset can be or comprise, for example, electronic eyewear, electroniceyeglasses, electronic headwear, an electronic helmet or partial helmet,an electronic visor, electronic goggles, or other type of ARD that canbe worn by a user.

The ARD 102 can comprise a sensor component 104 that can include a setof sensors that can sense or detect objects, people, conditions, events,or properties in an environment (e.g., in an area in proximity to asensor). The sensor component 104 can comprise, for example, a globalposition system (GPS) sensor or device, an orientation sensor(s),inertial accelerometer(s), and/or other types of sensor(s), to sense thelocation, movement, orientation, and/or position of the ARD 102 andassociated user and/or to sense the user interacting with or wearing theARD 102. The sensor component 104 also can comprise one or moreenvironmental sensors, such as, for example, a temperature sensor (e.g.,which can include infrared (IR) temperature sensor), a humidity sensor,an air pressure sensor, a chemical sensor, and/or another sensor(s).

The ARD 102 also can comprise an interface component 106 that canprovide a set of interfaces that can enable a user to interface orinteract with the ARD 102. The interface component 106 can receiveinformation (e.g., user input information) from the user via voicecommunication (e.g., voice command, search queries made by voice, inputof data via voice, . . . ) of the user (e.g., via an audio sensorcomponent, such as a microphone), via eye, facial, or body gestures(e.g., eye movement or focus, facial movement or expressions, hand orfinger gestures, . . . ) of the user in relation to the interfacecomponent 106, and/or via manipulation, by the user, of keys on akeyboard or keypad, or other buttons or controls of the interfacecomponent 106. The interface component 106 also can include a displayscreen(s) that can present visual images (e.g., photographs, video),images from other parts of the electromagnetic spectrum (e.g. infrared(IR), ultraviolet (UV), microwave, or any other zone across theelectromagnetic spectrum from radio waves to x-rays), textualinformation, and/or other information to the user, wherein such imagesor information can comprise augmented reality information and/or virtualreality generated and presented via the interface component 106 tofacilitate augmenting the reality of the user with respect to theenvironment associated with the user, enhancing the user experience ofthe user, enhancing the activities and/or productivity of the user,providing assistance and/or or training to the user, and/or enhancingthe safety and security of the user, etc.

The interface component 106 also can provide information and/orfeedback, including haptic feedback, to the user who is interacting withor wearing the ARD 102. In some implementations, the haptic feedback canfacilitate causing the ARD 102 to vibrate to provide an indication,alert, or notification to the user, which such indication, alert, ornotification can, for example, indicate that data (e.g., a message) hasbeen received or indicate that a hazardous condition is in proximity tothe user.

The ARD further can include an information management component (IMC)108 that can control operations of the ARD 102 and its variouscomponents (e.g., sensor component 104, interface component 106, . . .), and can generate augmented reality information and/or otherinformation (e.g., virtual reality information, user-relatedinformation, metadata, . . . ). For instance, the IMC 108 can analyzesensor information relating to conditions detected in an environment bythe sensor component 104 and/or other data that can be obtained from oneor more data sources, wherein the sensor component 104 can communicatethe sensor information to the IMC 108. Based at least in part on theresults of the analysis of the sensor information and/or the other data,the IMC 108 can determine and generate customized information,comprising augmented reality information and/or virtual realityinformation, that can be useful and/or relevant (e.g., pertinent) to theuser of the ARD 102, in accordance with defined information managementcriteria. The IMC 108 can present or facilitate presenting, via theinterface component 106, the customized information to the user.

The IMC 108 can be associated with (e.g., communicatively connected to)the sensor component 104 and the interface component 106 via a buscomponent 110, which can provide one or more buses (e.g., system bus)that can couple (e.g., communicatively connect) various components ofthe ARD 102.

The ARD 102 also can comprise a processor component 112 that can operatein conjunction with the other components (e.g., sensor component 104,interface component 106, IMC 108, bus component 110, data store 114, . .. ) to facilitate performing the various functions and operations of theARD 102. The processor component 112 can employ one or more processors(e.g., central processing units (CPUs), graphical processing units(GPUs), field-programmable gate arrays (FPGAs), etc.), microprocessors,or controllers that can process data, such as data relating to a user ofthe ARD 102, data relating to an environment in which the user and theARD 102 are located, maintenance-related data, safety-related data,security-related data, design-related data (e.g., data relating todesign of an industrial automation system, or data relating to design ofa product), human-machine interface (HMI) design-related data,asset-related data (e.g., device data, process data, asset data, systemdata, etc.) associated with systems (e.g., industrial automationsystems) or products, customer or client related data, data relating toparameters associated with systems or products, algorithms (e.g.,algorithm(s) relating to recognizing or identifying respectiveindustrial devices, industrial processes, industrial assets,network-related devices, interrelationships between such devices,processes, or assets), etc., to facilitate performing operations andtasks (e.g., in connection with an industrial automation system(s), orin connection with another type of environment); and can control dataflow between the ARD 102 and other components or devices associated withthe ARD 102.

In accordance with various aspects, the ARD 102 can include the datastore 114. The data store 114 can store data structures (e.g., userdata, metadata); code structure(s) (e.g., modules, objects, classes,procedures), commands, or instructions; data relating to a user of theARD 102, data relating to an environment in which the user and the ARD102 are located, maintenance-related data, safety-related data,security-related data, design-related data (e.g., data relating todesign of an industrial automation system, or data relating to design ofa product), HMI design-related data, asset-related data (e.g., devicedata, process data, asset data, system data, etc.) associated withsystems (e.g., industrial automation systems) or products, customer orclient related data, data relating to parameters associated with systemsor products, algorithms (e.g., algorithm(s) relating to recognizing oridentifying respective industrial devices, industrial processes,industrial assets, network-related devices, interrelationships betweensuch devices, processes, or assets), and so on. The processor component112 can be functionally coupled (e.g., through a memory bus of the buscomponent 110) to the data store 114 in order to store and retrieve datadesired to operate and/or confer functionality, at least in part, to thesensor component 104, interface component 106, IMC 108, etc., of the ARD102 and/or substantially any other operational aspects of the ARD 102.

It is to be appreciated and understood that the various components ofthe ARD 102 can communicate data, instructions, or signals between eachother and/or between other components associated with the ARD 102 asdesired to carry out operations of the ARD 102. It is to be furtherappreciated and understood that respective components (e.g., the sensorcomponent 104, interface component 106, IMC 108, etc.) of the ARD 102can be a stand-alone unit (e.g., attached or attachable to the ARD 102),can be included within the ARD 102 (as depicted), can be incorporatedwithin another component of the ARD 102, can be a component separatefrom the ARD 102, and/or can be virtually any suitable combinationthereof, as desired.

Referring to FIG. 2 (along with FIG. 1), FIG. 2 depicts a block diagramof an example sensor component 104 that can comprise a set of sensorsthat can be used in conjunction with the ARD 102, in accordance withvarious embodiments and aspects of the disclosed subject matter. Thesensor component 104 can include an image capture component 202 (e.g.,camera component) that can include one or more cameras that can be usedcapture visual images (e.g., photographs, video comprising video images)or images from other parts of the electromagnetic spectrum (e.g., IR,UV, microwave, or any other zone across the electromagnetic spectrumfrom radio waves to x-rays). The one or more cameras of the imagecapture component 202 can comprise one or more out-facing cameras thatcan have a camera lens(es) that is (or are) facing away from the userassociated with (e.g., wearing) the ARD 102 and/or one or moreuser-facing cameras that can have a camera lens(es) that is (or are)facing towards the user associated with the ARD 102. The one or moreout-facing cameras can capture images in the field of view of the userof the ARD 102 and/or images that are outside the field of view of theuser of the ARD 102 (e.g., behind the user, or outside of the direct orperipheral view of the user). The one or more out-facing cameras alsocan capture images of gestures of the user (e.g., physical gesturesemployed by the user to generate commands or requests directed to theARD 102), biometric information or features (e.g., fingerprint orhandprint) of the user, bodily dimensions of the user (e.g., dimensionsof hand or other body part of the user), movement-related features(e.g., walking gait) of the user, etc.

The one or more user-facing cameras (e.g., one or more inward-facingcameras) can capture images regarding the user, such as, for example,images of the facial expression of the user (e.g., to facilitatedetermining or identifying mood, impairedness, consciousness, or othercharacteristics or conditions of the user), facial or visual (e.g., eye)gestures of the user (e.g., employed by the user to initiate faciallygenerated commands or requests directed to the ARD 102), images of theeye focus or line-of-sight of the user, and/or images of the biometricfeatures (e.g., iris, retina, facial features or dimensions, . . . ) ofthe user. The images captured by the one or more cameras of the imagecapture component 202 can be multi-dimensional images (e.g.,two-dimensional images, three-dimensional images).

The sensor component 104 also can comprise an audio sensor component 204(e.g., microphone component) that can comprise one or more microphonesor audio sensors that can receive or sense audio signals (e.g., voice ofthe user, voice of another user, or miscellaneous other types ofsounds). For instance, the audio sensor component 204 can be employed toreceive voice communications from the user of the ARD 102, wherein thevoice communications can comprise commands or requests (e.g., requestfor additional augmented reality information) directed to the ARD 102,authentication information (e.g., voice characteristics, passcode, oranswer to security question, . . . ) provided by the user, feedbackinformation (e.g., observations, factual information, or parameters ofconditions of an area, system, product, part, asset, or process made bythe user) directed to the ARD 102, and/or information the user desiresto be stored by the ARD 102. The audio sensor component 204 also cansense sounds in the environment, wherein the sounds can indicate theoccurrence of an event or a hazard (e.g., gun shots, explosion, accident(e.g., vehicle accident), mechanical problem, . . . )

The sensor component 104 can include a location component 206 that canemploy one or more location sensors to sense or determine the locationof the ARD 102 and associated user and/or sense or determine therespective locations of respective objects (e.g., people, furniture,buildings, structures, appliances, machines, devices, trees, plants,hills, signs, poles, wires, . . . ) in an environment (e.g., in anarea), including sensing the relative location of respective objectsand/or the user to each other. The location component 206 can comprise,for example, a GPS system or device, an accelerometer, a gyroscope,and/or another component that can determine the respective locations ofthe ARD 102, the user, and/or other objects.

The sensor component 104 can comprise an orientation component 208 thatcan sense, determine, and/or measure the orientation of the ARD 102, andaccordingly the orientation of the user associated with (e.g., wearing)the ARD 102, relative to the environment. The orientation component 208can comprise one or more accelerometers, gyroscopes, inertialmeasurement unit (IMU) devices, and/or other sensors that can sense,determine, and/or measure the acceleration, position, motion, and/ororientation of the ARD 102 relative to the environment. Theaccelerometer can be, for example, a single axis or a multi-axes (e.g.,two-axes or three-axes) accelerometer that can sense, determine, and/ormeasure acceleration along one, two, or three axes, wherein suchaccelerometer data can be used to determine or measure orientation andmotion of the ARD 102. As another example, the gyroscope can be a singleaxis or a multi-axes (e.g., two-axes or three-axes) gyroscope that cansense or measure rotation along one, two, or three axes, wherein suchgyroscope data can be used to determine or measure orientation and/ormotion of the ARD 102. An IMU device can employ one or more sensors,such as an accelerometer(s), a gyroscope(s), and/or a magnetometer(s).

The sensor component 104 further can include a scanner component 210that can scan objects or images to facilitate identifying objects orinformation associated with objects or images. The scanner component 210can comprise a radio frequency identification (RFID) reader that canscan RFID tags associated with the user, other entities, or objects(e.g., product parts, industrial assets, . . . ) in proximity to theRFID reader to obtain identification information from the RFID tags tofacilitate identifying the user, other entities, or objects. The scannercomponent 210 also can include a bar code reader that can read bar codeinformation contained in bar codes associated with objects (e.g.,product parts, industrial assets, . . . ) or entities to facilitateidentifying the objects or entities.

The sensor component 104 also can comprise a recognition component 212that can employ one or more recognition techniques to sensecharacteristics, features, or conditions of entities or objects tofacilitate recognizing or identifying entities or objects. Therecognition techniques that can be utilized by the recognition component212 can include, for example, pattern recognition, object recognition,facial recognition, voice recognition, audio recognition, imagerecognition, spatial relationship recognition, animal recognition, poserecognition, action recognition, shape recognition, scene recognition,behavior recognition, sound recognition, scent recognition, motionrecognition, hue recognition, feature recognition, edge recognition,texture recognition, timing recognition, location recognition, and/orany other suitable recognition technique) to facilitate determiningcharacteristics, features, or conditions of the entities or objects, andrecognizing or identifying the entities or objects based at least inpart on the sensed characteristics, features, or conditions of entitiesor objects.

The sensor component 104 can comprise an environment component 214 thatcan employ one or more sensors to sense various conditions orcharacteristics of an environment in proximity to or otherwiseassociated with the ARD 102. The one or more sensors (e.g., environmentsensors) can include, for example, a temperature sensor (e.g., IRsensor, thermal imaging sensor, semiconductor-based temperature sensor,resistance temperature detector (RTD), . . . ), a humidity sensor, abarometer, an altimeter, a chemical sensor, a radiation sensor, abiohazard sensor, a smoke detector, an electromagnetic field sensor, ascent sensor, a pressure sensor, a moisture sensor, and/or a lightsensor.

The temperature sensor can detect or measure a temperature level of anobject (e.g., outside of object, inside of object) or area. The humiditysensor can detect, measure, determine, or facilitate determining thehumidity level of the environment in proximity to the ARD 102 (e.g., inproximity to the humidity sensor of the ARD 102). The barometer cansense, measure, determine, or facilitate determining the air pressurelevel of the environment in proximity to the ARD 102 (e.g., in proximityto the barometer of the ARD 102). The altimeter can detect, measure,and/or determine an altitude of the ARD 102 or an object in proximity tothe ARD 102.

The chemical sensor can detect, measure, and/or facilitate identifyingchemical elements or information in the environment in proximity to theARD 102. The radiation sensor can detect, measure, and/or facilitateidentifying radiation, including an amount or a type of radiation, inthe environment in proximity to the flow management light (e.g., inproximity to the radiation sensor of the flow management light). Thebiohazard sensor can detect, measure, and/or facilitate identifying abiohazardous condition in the environment (e.g., in the air of theenvironment or emitted by an object(s) in the environment) in proximityto the biohazard sensor of the ARD 102. The smoke detector can detectsmoke or other air impurities, or measure smoke or other air impurities,in the environment in proximity to the smoke detector of the ARD 102.The electromagnetic field sensor can sense or measure electromagneticfields in the environment in proximity to the ARD 102 (e.g., inproximity to the electromagnetic field sensor of the ARD 102). The scentsensor can sense and/or facilitate identifying scents in the environment(e.g., in the air of the environment or emitted by an object(s) in theenvironment) in proximity to the scent sensor of the ARD 102.

The pressure sensor can detect or measure pressure (e.g., an amount ofpressure) in the environment in proximity to the pressure sensor of theARD 102. The moisture sensor can detect an amount of moisture in theenvironment (e.g., in the air of the environment or emitted by anobject(s) in the environment) in proximity to the moisture sensor of theARD 102. The light sensor can detect or measure light or an amount oflight in proximity to the light sensor of the ARD 102.

The sensor component 104 further can comprise one or more other sensors216 that can monitor and detect various characteristics, features,and/or conditions related to entities or objects to facilitatedetermining various characteristics, features, conditions, qualities,etc., of the entities or objects. Such other sensors can include, forexample, a motion sensor, a radar sensor, an ultrasound imaging sensoror device, a light detection and ranging (LIDAR) sensor, a soundnavigation and ranging (SONAR) sensor, or a microwave sensor.

The motion sensor can sense movement and/or direction of movement of anobject(s) or entity(ies) in proximity and relation to the motion sensorof the ARD 102. The radar sensor can employ radar technology (e.g.,Doppler radar) to facilitate detecting an object(s) or entity(ies),including detecting the location and/or movement of an object(s) orentity(ies), in proximity to the radar sensor of the ARD 102.

The ultrasound imaging sensor can employ ultrasound technology to detectfeatures of or objects or entities in an area in proximity to theultrasound imaging sensor of the ARD 102, and can facilitate generatingimages (e.g., ultrasound images) that can represent the features of orobjects or entities in the area in proximity to the ARD 102. The LIDARsensor can employ a laser light (e.g., a pulsed laser light) to detectfeatures of or objects or entities in the area in proximity to the LIDARsensor of the ARD 102, and can facilitate generating images (e.g., LIDARimages) that can represent the features of or objects or entities in thearea in proximity to the ARD 102. The SONAR sensor can employ ultrasoundtechnology to detect features of or objects or entities in the area inproximity to the ARD 102, and the distance between the features orobjects or entities and the ARD 102, and can facilitate generating ofimages (e.g., SONAR images) that can represent the features of orobjects or entities in the area in proximity to the SONAR sensor of theARD 102. The microwave sensor can employ microwaves to facilitatedetecting objects or entities, including the movement of objectsentities, in the area in proximity to the microwave sensor of the ARD102.

Turning to FIG. 3 (along with FIG. 1), FIG. 3 illustrates a blockdiagram of an example interface component 106 that can comprise a set ofinterfaces that can be used in conjunction with the ARD 102, inaccordance with various embodiments and aspects of the disclosed subjectmatter. The interface component 106 can comprise a display component302, an audio component 304, a haptic component 306, and a key/controlcomponent 308 (e.g., keyboard, keypad, and/or control buttons). Theinterface component 106, including its various components (e.g., 302,304, 306, 308) can be associated with (e.g., communicatively connectedto) the IMC 108.

The display component 302 can comprise or generate one or more displayscreens of desired size and shape that can be situated or presented inthe field of view of the user of the ARD 102 (e.g., situated orpresented on or near the lens or lenses of the ARD 102). The one or moredisplay screens of the display component 302 can present data, such ascustomized data (e.g., augmented reality information), which can be datathat is automatically or dynamically generated by the IMC 108 and/orcommunicated to the display component 302 by the IMC 108, and/or whichcan be data requested by the user. The IMC 108 can determine (e.g.,automatically and/or dynamically determine) a location(s) on a displayscreen of the display component 302 where the data is to be presented(e.g., displayed), based at least in part on the defined informationmanagement criteria.

The audio component 304 can comprise audio generation and/orpresentation mechanisms to facilitate generating, amplifying, and/oremitting sounds (e.g., voices, music, audio alerts or notifications, . .. ) from the audio component 304. For instance, the audio component 304can comprise one or more amplifiers that can amplify audio signals toachieve a desired volume level, one or more speakers that can emit thesounds (e.g., processed or amplified sounds), and/or sound processingmechanisms or devices (e.g., digital signal processor, analog-to-digitalconverter, digital-to-analog converter, encoder, decoder, soundequalizer, . . . ) that can process audio signals to generate processedaudio signals. In some implementations, the audio component 304 also canbe associated with or can include the audio sensor component (e.g.,microphone(s)) of the sensor component 104 to receive sounds (e.g.,voice commands or other speech or sounds) from the user or other soundsaudible in the area where the audio component 304 is located. The audiocomponent 304 can receive audio information (e.g., data (e.g., augmentedreality data), alerts, notifications, indicators, etc., in audio form)from the IMC 108 for presentation to the user.

The haptic component 306 (e.g., feedback component) can providefeedback, including haptic feedback, to the user who is interacting withor wearing the ARD 102. The haptic feedback can comprise hapticinformation, communications, or signaling in any of its various forms.In some implementations, the haptic component 306 can be or can comprisea vibration component that can cause the haptic component 306 or otherpart of the ARD 102 to vibrate to provide an indication, alert, ornotification to the user (e.g., relating to an issue or data).Additionally or alternatively, the haptic component 306 can employ oneor more other types of haptic feedback (e.g., haptic communications orsignaling). For example, the haptic component 306 can apply forces ormotion to the user to provide a tactile sensation(s) to the user, whichcan facilitate providing an indicator, alert, or notification to theuser. Some examples of other types of haptic feedback can comprise, forinstance, feedback that can be, or can simulate, a brush on the skin ofthe user, a waft of air that can be perceived or felt by the user, asensation of heat or cold on the skin of the user (e.g., in connectionwith a hazard relating to hot or cold temperatures). The hapticcomponent 306 can communicate respective (e.g., different) types offeedback (e.g., respective types of vibrations or other haptic feedback)for respective types of indications, alerts, or notifications.

For example, the haptic component 306 can communicate a first type offeedback (e.g., a first type of vibration, or other first type of hapticfeedback, of the haptic component 306) that can alert the user to ahazardous condition of which the user is to be made aware (e.g., becausethe user is in proximity to the hazardous condition, or so the user cantake an action responsive to the hazardous condition). As anotherexample, the haptic component 306 can communicate a second type offeedback (e.g., a second type of vibration, or other second type ofhaptic feedback, of the haptic component 306) that can notify the userthat a message (e.g., an email message, a text, instant, or multimediamessage, or a social media message) to the user has been received (e.g.,because the user is in proximity to the hazardous condition, or so theuser can take an action responsive to the hazardous condition). As stillanother example, the haptic component 306 can communicate a third typeof feedback (e.g., a third type of vibration, or other third type ofhaptic feedback, of the haptic component 306) that can notify or alertthe user that an event has occurred. The haptic component 306 cancommunicate other types of feedback (e.g., other types of vibratorysignals, and/or other types of haptic feedback signals besides vibratorysignals) to the user with regard to other types of indications, alerts,or notifications. As some examples involving vibratory signals, thetypes of vibrational feedback can comprise, for example, a singlevibration, two distinct vibrations in succession, three distinctvibrations in succession, a single long vibration, a single shortvibration that can be shorter in duration than the single longvibration, a combination of short vibration(s) and long vibration(s), orany desired variation thereof.

The key/control component 308 can comprise a keyboard (e.g., a keyboardhaving keys representing alphanumeric characters, symbols, orfunctions), a keypad (e.g., a keypad having keys representingalphanumeric characters, symbols, or functions), and/or one or morecontrol buttons that can represent respective controls, functions, alphanumeric characters, or symbols. In some implementations, the keyboard,keypad, or control buttons can be part of a unit that can be held oraccessed by the user to facilitate selection of the keys or controlbuttons by the user, wherein the unit can be connected via a wireline orwireless connection to the IMC 108, processor component 112, and/orother components of the ARD 102. In other implementations, keyboard,keypad, or control buttons can be displayed on a display screen of thedisplay component 302, wherein keys or control buttons can be selectedvia visual cues (e.g., eye movements), audio cues (e.g., spoken word),gestures, or another type of action(s) of the user of the ARD 102. Instill other implementations, the image capture component 202 of thesensor component 104 can capture gestures (e.g., eye gestures, handgestures, . . . ) of the user to facilitate selecting keys or controlbuttons, wherein the image capture component 202 can interact with thekey/control component 308 and/or another component(s) to facilitateconverting the gestures into selection of keys or control buttons.

It is to be appreciated and understood that the image capture component202, the audio sensor component 204, and/or one or more other sensors ofthe sensor component 104 also can be part of the interface component106.

With further regard to FIG. 1 (along with FIGS. 2 and 3), the IMC 108 ofthe ARD 102 can analyze and process various inputs of information (e.g.,sensor information, information from other data sources), such as, forexample, data relating to location, movement, or orientation of the useror the ARD 102, object recognition of objects within the field of viewof the user or the ARD 102, data input received or obtained from theuser by the ARD 102 (e.g., via the sensor component 104 or the interfacecomponent 106). Based at least in part on the results of the analysisand processing of the various inputs of information, the IMC 108 cangenerate and present (e.g., display or overlay) customized information(e.g., augmented reality information and/or virtual reality information)that can be relevant (e.g., pertinent) and useful to the user while theuser is performing tasks (e.g., tasks relating to an industrialautomation system or relating to another type of system or environment).Such customized information can be presented to the user via theinterface component 106 (e.g., a display screen(s) of the displaycomponent 302, audio component 304, and/or haptic component 306. Suchcustomized information can comprise, for example, photographs, video,computer aided design or drafting (CAD) objects, tags, dials, alerts,and/or notifications, etc., that can be relevant or useful to the userin connection with the tasks (e.g., work tasks) or activities the useris engaging in or is preparing to engage in and/or the environment theuser is in or a remote environment with which the user is interacting.

In some implementations, with the user (e.g., operator) interacting withor wearing the ARD 102, the ARD 102 can assist the user in a number ofways, such as disclosed herein, to improve the speed of the user and ofthe speed of assembly or production of products, the quality of assemblyor production of products, the security of assembly and production ofproducts, and the safety of the user and the environment in theperformance of tasks relating to assembly of products by the user. TheARD 102 can advise or instruct the user through visual signals (e.g.,via the display component 302), audible signals (e.g., via the audiocomponent 304), and/or haptic feedback (e.g., via the haptic component306) to facilitate enhancing the assembly process (e.g., enhancing, thespeed, quality, security, and safety associated with the assemblyprocess).

In some implementations, the disclosed subject matter, by employing thefeatures of the ARD 102, can capture enhancements to tasks or activitiesand provide assistance in the performance of tasks or activities (e.g.,work tasks or activities, leisure activities, . . . ) in areas, such as,for example, 1) remote control, 2) route selection, and 3) life coach,butler, or tour guide.

The ARD 102 can employ methods, techniques, features, and activities tosolve problems relating to remote control, route selection, and lifecoach, butler, or tour guide. For instance, the ARD 102 can employ thesensor component 104 to sense location, orientation, and/or motionrelated information associated with the ARD 102. Based at least in parton the results of analyzing the sensor information relating to location,orientation, and/or motion of the ARD 102, the IMC 108 can determine orresolve the location, orientation, and/or motion (e.g., movement) of theARD 102 with respect to an assembly. The sensor component 104 also cansense conditions (e.g., environmental conditions) associated with theenvironment where the user and associated ARD 102 are located. The IMC108 can analyze the sensor information relating to the conditionsassociated with the environment. Based at least in part on the resultsof analyzing the sensor information relating to the conditions of theenvironment, and/or the location, orientation, and/or motion of the ARD102, the IMC 108 can determine customized data (e.g., augmented realityinformation and/or virtual reality information) that can be relevant(e.g., pertinent) and useful to the user. The IMC 108, via the displaycomponent 302, can present (e.g., overlay) the customized data on thedisplay screen(s) of the display component 302 of the ARD 102. Thecustomized data can comprise augmented reality information (e.g., realtime information relating to the environmental conditions) and/orvirtual reality information (e.g., virtual view objects that canrepresent objects associated with the environment). The ARD 102 also canwork with other networks and/or systems (e.g., industrial, commercialand corporate computer networks and/or systems) as well as databases andservers to exchange data and/or process the tasks (e.g., assembly tasks)that are done to accomplish the described goal or solution (e.g.,assemble a product).

There can be instances when there can be some danger or at leastpotential danger in a user physically going to a cabinet (or otherobject or device) to perform a task (e.g., adjusting a bit or control(e.g., toggling or forcing a bit or control)) on a component of thecabinet. The disclosed subject matter can overcome these and otherissues by employing various techniques and features, as describedherein.

With regard to some of the remote control features of the disclosedsubject matter, the user can employ the ARD 102 to perform (e.g.,remotely perform) tasks (e.g., adjusting a bit or control) on acomponent of the cabinet from a safer, remote distance from the cabinet,instead of the user physically being by the cabinet to perform the taskson the component of the cabinet. That is, using the ARD 102 to performthe tasks on the component of the cabinet, the user can be a desireddistance away (e.g., 10 feet away, 10,000 km, . . . ) away from thecabinet when performing the tasks (e.g., using the ARD 102) inconnection with the cabinet. For example, the user can use the ARD 102to communicate with the component (or an associated component) of thecabinet via a wireline or wireless communication connection. The ARD 102can receive information relating to settings (e.g., bit or controlsettings) of the component of the cabinet from the cabinet. The IMC 108can instruct the interface component 106 (e.g., display component 302)to present the information relating to the settings of the component tothe user (e.g., on a display screen). The IMC 108 also can facilitatepresenting (e.g., via the display screen of the display component 302)virtual settings (e.g., virtual bits or controls) for the user to view.The user can decide to perform a task to adjust the settings. In thatregard, the user can employ keys, buttons, or controls on the interfacecomponent 106 (e.g., key/control component 308 of the interfacecomponent 106) and/or voice commands (e.g., via the interface component106) to adjust the settings (e.g., by adjusting the virtual bits orcontrols) of the component of the cabinet from a location at a remote orsafe distance from the cabinet. The settings of the component of thecabinet can be adjusted accordingly, in response to the adjustment tothe settings of the component performed by the user using the ARD 102.

In some implementations, the IMC 108 can generate and facilitatepresenting, via a display screen of the interface component 106, an HMI(e.g., a virtual HMI), or the ARD 102 can be associated with (e.g.,communicatively connected to) an HMI that can be presented to the uservia the ARD 102. The HMI can be virtual and can be used (e.g.,manipulated) by the user, using the IMC 108 and interface component 106of the ARD 102, to control assets (e.g., devices, machines, processes, .. . ) from a suitable (e.g., safe) distance away from the assets.

With regard to other aspects of remote control using the ARD 102, theuser can use the ARD 102 to facilitate performing assembly, maintenance,and/or design, deployment, and/or configuration tasks by controlling arobot (e.g., robotic device) remotely via the ARD 102. For example, theuser can use the ARD 102 (e.g., the IMC 108 and/or other components ofthe ARD 102) to control operation of a drone to carry a robotic device(e.g., an anthropomorphic robot) to a remote (e.g. a supervisory controland data acquisition (SCADA)) location or a dangerous location (e.g., alocation of an oil pump). The user also can use the ARD 102 (e.g., theIMC 108 and/or other components of the ARD 102) to control the roboticdevice virtually, e.g., via gestures or commands of the user that arereceived by the interface component 106 and/or sensor component 104 ofthe ARD 102 from the user. The procedures employed by the ARD 102 and/orrobotic device for performing the tasks by the ARD 102 and/or roboticdevice can either be live or can be previously developed at a more localor on premises installation.

With regard to route selection for travel by a user, the ARD 102 canenhance route selection associated with the user (e.g., route selection(e.g., smarter GPS, or foot traffic version of driverless cars)). TheARD 102 can utilize GPS, maps, and/or object recognition to guide theuser as the user walks to the destination. Also, the ARD 102, employingthe sensor component 104 and IMC 108, can detect, identify, and useinformation (e.g., visible, perceived, or sensed) information abouthazards (e.g., heat, voltage, moving machines, precipices, . . . ) indatabases and in real-time to enable the user to efficiently and safelystay on a desirable track to the destination. The ARD 102, employing theinterface component 106 and IMC 108, also can generate and presentalerts or notification messages to warn the user about these hazards,stairs, and/or other moving objects or people while the user travels enroute to the destination.

For instance, the user can instruct the ARD 102 regarding a destinationwhere user wants to go. The ARD 102 can use sensors (e.g., locationsensor, orientation sensor, motion sensor, GPS, and/or object sensors, .. . ) of the sensor component 104 and/or other data sources (e.g., mapor travel data sources) to determine the origination location of theuser, determine the destination location of the user, determine objects,obstacles, and/or hazards in the area between the origination locationand destination location, and/or determine other relevant information inconnection with the travel by the user. Based at least in part on theresults of analyzing the travel instructions (e.g., originationlocation, destination location) received from the user, the sensorinformation received from the sensor component 104, and/or othertravel-related data (e.g., map or travel related data) received from theother data sources, the IMC 108 can determine a desired (e.g., suitable,acceptable, enhanced, or optimal) travel route for the user to traveland can provide instructions to the user, via the interface component106, to guide the user from the origination location to the destinationlocation in a safe, secure, and efficient manner. The guidance regardingthe travel route the ARD 102 provides to the user can be visual guidance(e.g., visual display of the directions along the travel route,including streets or landmarks along or associated with the travelroute, and/or textual travel directions), verbal guidance (e.g., anaudio presentation of the directions along the travel route, and/orhaptic feedback (e.g., a haptic indicator to indicate a user is to makea change of direction; another haptic indicator to indicate that a userhas missed a change of direction; or still another haptic indicator toindicate that the travel route has been changed or updated). The ARD102, via the interface component 106, can provide such guidanceregarding the travel route on a continuous, periodic, orevent-occurrence basis as the user proceeds along the travel route.

As part of determining the travel route between the origination locationand the destination location and/or advising or guiding the user as theuser proceeds along the travel route, the IMC 108 can detect, determine,and/or identify objects, obstacles, and/or hazards (or potentialhazards) in the area between the origination location and destinationlocation based at least in part on the sensor information and/or othertravel-related information obtained from the other data sources. The IMC108 can determine, customize, and/or tailor the travel route based atleast in part on the objects, obstacles, and/or hazards (or potentialhazards) in the area between the origination location and destinationlocation, including structuring the travel route to avoid or at leastsubstantially avoid (e.g., mitigate, reduce, minimize, or eliminate)obstacles, and/or hazards (or potential hazards) along the path the usertravels and/or to warn (e.g., via a visual, audio, and/or haptic warningindicator(s)) the user of any obstacles and/or hazards (or potentialhazards) the user may face while traveling along the determined travelroute.

If the travel route includes any objects, obstacles, and/or hazards (orpotential hazards), the IMC 108 can facilitate presenting, via theinterface component 106 (e.g., on the display screen) of the ARD 102, anindicator or notification of, and/or can highlight, the object,obstacle, or hazard to warn or notify the user of such object, obstacle,or hazard before, but when the user is in proximity to, such object,obstacle, or hazard.

As the user travels along the travel route, the sensor component 104 cancontinue to collect additional sensor information, and/or the other datasources can continue to collect additional travel-related information,and the IMC 108 can, continuously, periodically, and/or on anevent-occurrence basis, analyze the additional sensor information and/orthe additional travel-related information. Based at least in part on theresults of the analysis of the additional sensor information and/or theadditional travel-related information, the IMC 108 can determine whetherthe user is still on course on the travel route, whether the travelroute is to be changed, whether a new obstacle or new hazard (or newpotential hazard) has appeared along the travel route, whether anindicator or notification (or highlighting) is to be presented to theuser, via the interface component 106, to warn or notify the user of anobject, obstacle, or hazard associated with the travel route.

For example, as the user travels along the travel route, the sensorcomponent 104 can detect when a newly occurring or newly recognizedhazard or obstacle (e.g., a “Don't Walk” sign, stairs, moving people orobjects, . . . ) occurs. The IMC 108 can analyze the sensor data fromthe sensor component 104 and can determine an indicator, notification,and/or highlighting to present to the user and/or an adjustment to thetravel route to present to the user, via the interface component 106.

Employing these and other aspects of the ARD 102, the ARD 102 also hasthe capability to be used by blind and visually impaired people (e.g.firefighters or other emergency responders in a dark or smoky building)to enhance their performance of tasks, and enhance their safety andsecurity. The ARD 102, employing the sensor component 104 and/orinformation obtained from another data source, can monitor and detectnatural hazards, such as, for example, streams, avalanche, cliff, holesin the ground, uneven or rocky surfaces, trees, or plants. The ARD 102,employing the sensor component 104 and/or information obtained fromanother data source, also can monitor, detect, and/or identify otherhazards or obstacles, such as, for example, open or closed doors (e.g.subway doors, building doors, . . . ), people approaching a door(including on the other side of a glass door), traffic signals such as“walk/don't walk” or traffic lights (e.g., green light, red light),curbs, hostile people with weapons, stairs, wires (e.g., downedelectrical wires, hanging wires, cable wires), walls, doors, furniture,barriers, machines, devices, assets (e.g., industrial assets),appliances, buildings, structures, building fixtures, lighting fixtures,railings, rails, logs, tracks, ditches, wells, fences, hills, vehicles,etc. The sensors of the sensor component 104 can employ objectrecognition techniques, edge detection techniques, or other recognitiontechniques to facilitate detecting and determining shapes of objects(e.g., persons, entities, obstacles, hazards, barriers, . . . ),dimensions of objects, and/or other features of objects.

The IMC 108 can determine an appropriate indicator, notification, and/orhighlighting to use to notify or warn the blind or visually impaireduser of such hazards or obstacles. For instance, the IMC 108 candetermine that an audio message and/or haptic feedback can be presented,via the interface component 106, to the user to notify or warn the userof the hazard or obstacle, to provide information regarding the hazardor obstacle (e.g., curb is 8 feet (or three steps) in front of you,street intersection is 20 feet ahead, pedestrian signal is 10 feet aheadand has changed from “walk” to “don't walk”), and/or to alter the travelroute of the user.

Additionally or alternatively, with regard to a user (e.g. firefightersor other emergency responders in a dark or smoky building) whose visionhas been impaired by a hazard, the IMC 108 can determine othercustomized information (e.g., augmented reality information and/orvirtual reality information) that can be presented, via the interfacecomponent 106 (e.g., via a display screen), to the user. For example,the IMC 108 can determine that a heat map indicating hot spots in theenvironment (e.g., areas of a building with higher temperatures), andinformation (e.g., virtual objects representing actual objects)indicating objects in the environment, can enhance the performance oftasks, and enhance the safety, of the user. The IMC 108 can generate aheat map of the environment based at least in part on sensor data of theenvironment from the sensor component 104. The IMC 108 also can generatevirtual objects that can represent actual objects and their relativeplacement (e.g., position) in the environment (e.g., in a room of thebuilding). The IMC 108 can facilitate presenting, via the display screenof the interface component 106, the heat map and the visual informationrelating to virtual objects in the environment to the user (e.g.,visually impaired user) to enhance the user's visual perception of theenvironment, enhance the performance of tasks by the user, and/orenhance the safety and security of the user.

The ARD 102 also can supplement or enhance information for a user who isdeaf and otherwise audibly impaired by translating key sounds intovisual information that can be perceived by the user. For instance, thesensor component 104 of the ARD 102 can monitor, detect, and/or identifypeople yelling, instructions being given by people, approachingvehicles, alarms, audible announcements, etc. The IMC can analyze thesensor data relating to people yelling, instructions being given bypeople, approaching vehicles, alarms, audible announcements, etc., andcan translate the audio signals relating to those events into visualsignals or information that can be perceived by the user via the displayscreen of the interface component 106.

In some implementations, the ARD 102 also can translate language forforeigners. For example, the IMC 108 of the ARD 102 can translate spokeninformation (e.g., sensed via the sensor component 104) into visual,translated text or audible, translated text. The sensor component 104also can sense written signs, menus, and/or cues. Based at least in parton the results of analyzing sensor data relating to the written signs,menus, and/or cues, the IMC 108 can translate the written signs, menus,and/or cues into corresponding textual information and/or audio signals.The IMC 108 can facilitate presenting, via the display screen and/orspeakers of the interface component 106, the translated text and/oraudio signals to the user.

The ARD 102 also can generate and provide cues, based at least in parton context (e.g., location of the user, camera input, . . . ) to localcustoms associated with the location. For instance, the sensor component104 (e.g., GPS or navigation system, camera, audio sensor, . . . ) canmonitor and detect the location of the user, and/or visual informationand/or audio information regarding the environment. Based at least inpart on the results of analyzing such sensor data regarding the locationand environment, the IMC 108 can determine the context relating to localcustoms or local items of interest for that location (e.g., country,region, . . . ), can determine cues or instructions that can enhance theactivities or actions of the user in that location, and can facilitatepresenting, via the interface component 106, the cues or instructions tothe user. The local customs and/or local items of interest can be or canrelate to, for example, which side of the road to drive on, how to greetpeople in that location, currency translation, food ingredients,politics and phrases to avoid while in that location, local history ofthat location, sights to see in or near that location, recent eventsassociated with that location (to start conversation), landmarks in ornear that location, mythologies associated with that location, and/orother local customs or items of interest.

In certain implementations, with regard to companionship, butler, tourguide, and life coach issues, the ARD 102 can be employed to provide anumber of services and information (e.g., customized information) basedat least in part on conditions or characteristics of the environmentdetected by the sensor component 104, information obtained from anotherdata source(s), and/or user instructions or preferences. The IMC 108 cananalyze the sensor data relating to the conditions or characteristics ofthe environment, other information from the other data source(s), and/orthe user instructions or preferences, and can determine information(e.g., customized information) to present to the user via the displaycomponent 302, the audio component 304, and/or the haptic component 306of the interface component 106.

As some examples, based at least in part on the results of suchanalysis, the IMC 108 can employ the interface component 106 to playcertain music, and/or present visual and/or audio versions of stories(e.g., frivolous stories, fables, local yore, teaching, history,personal (e.g., conglomeration of Facebook, Twitter, Instagram, Wiki(e.g., Wikipedia), . . . )). The IMC 108 also can employ the interfacecomponent 106 to inform the user or other people about the locale,people, places, and history associated with the location of the user;and/or generate and provide a list opportunities (e.g., here and nowopportunities), such as, for example, where to eat, what to do forentertainment, etc., wherein the IMC 108 can determine such informationbased at least in part on the analysis results. Additionally oralternatively, the IMC 108 can employ the interface component 106 topresent information (e.g., customized information) to teach the userabout abstracts: history, law, math, science, etc.; and/or teach theuser about the here and now: what is around the user now, who is aroundthe user now, etc., wherein the IMC 108 can determine such informationbased at least in part on the analysis results.

The IMC 108 also can determine and generate joke suggestions and/orspeech text, which can be presented to the user via the interfacecomponent 106 (e.g., the speech text can be presented via a teleprompterof the interface component 106). The ARD 102 also connect the user withhome, family, and/or friends as immersively as possible via text, voice,and/or video (e.g., 3-dimensional (3-D) video); remind the user ofsignificant events, things to do, appointments, gift preferences,manager type tasks, etc.; and/or allow connections and ability to act onideas: order flowers, adjust furnace, turn off water pump, balancecheckbook, complete tax return, perform production accounting and datareconciliation, etc., based at least in part on the results of the dataanalysis performed by the IMC 108.

In some implementations, the ARD 102 can inform the user when the useris becoming impaired (e.g., sleepy, inebriated, etc.). For example, thesensor component 104 can employ a camera(s) (e.g., user-facing camera ofthe interface component 106) to detect eye actions or facial expressionsof the user and/or an audio sensor to detect the speech of the user andpattern recognition. The IMC 108 can analyze (e.g., using patternrecognition on the visual sensor data or audio sensor data) the sensordata relating to the eyes, face, or speech of the user, and candetermine whether the user is impaired (e.g., sleepy, inebriated,unconscious, . . . ) or not impaired. In response to determining thatthe user is impaired, the IMC 108 can determine an indicator ornotification (e.g., visual, audio, and/or haptic indicator ornotification) to present to the user (or to another person) to notifythe user (or another person) that the user is impaired. The IMC 108 cancontrol the interface component 106 to have the interface component 106present the indicator or notification to the user and/or can facilitatecommunicating the indicator or notification to a device (e.g., anotherARD or other communication device) of another user.

In some embodiments, the ARD 102 can provide assembly advantages to theuser or other entity (e.g., assembly company) during an assembly processto assemble products or perform other installation work in connectionwith a product (e.g., article of manufacture, device, cabinet, unit,etc., comprising parts, components, wires, and/or circuitry, . . . ).The ARD 102 can be employed to overcome various problems that may occurduring assembly of products. For example, in connection with an assemblyprocess, there may be various problems regarding a wrong part beingused, a part being used in the wrong place, a part may be connected tothe wrong thing(s), a part may not be traceable (e.g., a serial numberof a part), a part may not meet incoming inspection criteria, parts fromassembly may be forgotten, a user may not be trained, a user may not beauthorized, a user may be in the wrong place or assembling the wrongunit, and/or assembly training materials may be insufficient. The ARD102 can employ methods, techniques, components, features, and activitiesthat can overcome these and/or other problems.

In some implementations, with the user (e.g., operator) interacting withor wearing the ARD 102, the ARD 102 can assist the user in a number ofways, such as disclosed herein, to improve speed of the user and ofassembly or production, quality of assembly or production, security ofassembly and production, and safety of the user and the environment inthe performance of tasks relating to assembly of products by the user.The ARD 102, employing the interface component 106, can advise orinstruct the user through visual signals, audible signals, and/or hapticfeedback to facilitate enhancing the assembly process (e.g., enhancing,the speed, quality, security, and safety associated with the assemblyprocess).

The ARD 102 can be employed to mitigate (e.g., reduce, minimize, oreliminate) wrong parts being used by the user (e.g., operator orassembler) during assembly of the product. The IMC 108 can be aware ofwhat parts are to be used, what order the parts are to be assembled, therespective locations where the parts are to be assembled or installed,etc., to facilitate desirable (e.g., efficient) assembly of the product.In some implementations, the ARD 102 can be used step by step during theassembly process to advise the user of the ARD 102 which part is next tobe assembled (e.g., connected, installed, . . . ) for the product andcan use feedback to enable the user to determine which part is next inthe assembly process. As an example of feedback, the IMC 108 cangenerate one or more indicators that can be presented via the interfacecomponent 106 to the user, wherein the one or more indicators cancomprise visibly highlighting the next part by coloration and/orblinking (e.g., a green colored and/or blinking box surrounding the nextpart or arrow pointing to the next part, as presented on the displayscreen of the display component 302; and/or a voice message emitted viaa speaker of the audio component 304 to the user). The ARD 102 also canemploy the interface component 106 (e.g., the display screen of thedisplay component 302) to show the parts of the product virtually on thedisplay screen over the live view the user has (e.g., the live view viathe lens component of the ARD 102).

For instance, the IMC 108 can receive the assembly information,comprising task-related information, relating to the product assemblyfrom the data store 114 and/or another data source (e.g., external datasource), the sensor component 104 (e.g., one or more sensors) and/or theuser. The IMC 108 can analyze the assembly information. Based at leastin part on the results of the analysis, the IMC 108 can determine thetask (e.g., the next task) relating to assembly of the product.

The assembly information can comprise directions and/or images relatingto assembling the product, including information regarding therespective parts of the product, the order of the assembling of parts ofthe product, respective parameters of respective parts, any notableconsiderations, conditions, or hazards associated with assembling theproduct, and/or other types of assembly information. Notableconsiderations, conditions, or hazards associated with assembling theproduct can comprise, for example, indicating whether a part orassembling (e.g., installing, connecting, . . . ) of the part involves apotential hazard (e.g., the part has a sharp edge and is to be handledwith care, or installation of the part involves high voltage and thepotential for electrical shock, so proper precautions should be taken)and/or indicating whether the part is fragile and should be handled withcare, among other types of considerations, conditions, or hazards thatmay exist.

The IMC 108 also can determine the part (e.g., next part) to beassembled as part of the assembly task based at least in part on theresults of analyzing the assembly information. The IMC 108 can generatecustomized information (e.g., augmented reality information and/orvirtual reality information), comprising part information, that canindicate the part to be assembled as part of the assembly task.

The IMC 108 can facilitate presenting, via the interface component 106of the ARD 102 associated with the user, the customized information,comprising the part information, to the user. The customizedinformation, comprising the part information, can include a visualpresentation regarding the part information (e.g., a drawing of the part(e.g., in space), and/or the part being visually highlighted oremphasized by color, by flashing, and/or by other emphasis means), forexample, as if the part is present (e.g., virtually present) in the liveview of the assembly of the product (e.g., partial assembly comprisingthe part) that can be viewable through the ARD 102 (e.g., displaycomponent 302) by the user, and/or can include audio informationregarding the part information presented via the interface component 106(e.g., speaker(s) of the audio component 304). Such customizedinformation can facilitate instructing the user as to which part toselect (e.g., choose) for assembly or installation at this point in theassembly process.

Referring briefly to FIG. 4 (along with FIG. 1), FIG. 4 depicts adiagram of various example visual presentations 400 in the field of viewof a user in connection with an assembly process, in accordance withvarious aspects and embodiments of the disclosed subject matter. Asdepicted in the visual presentations of FIG. 4, the IMC 108 canfacilitate presenting customized information on the display screen ofthe display component 302 associated with the lens component, whereinthe customized information can comprise a visual illustration of thepart 402, and optionally can include a visual emphasis 404, such as aring or rectangular shaped box around the part (e.g., to highlight thepart from other parts that may be viewable on the display screen).

In some implementations, the IMC 108 can determine the correct bin orpart storage location where the part is located based at least in parton an analysis of the assembly information and/or other information(e.g., stored in the data store 114 or an external data source) that canindicate the location of the correct bin or part storage location. TheIMC 108 also can determine the travel route (if any travel to get thepart is to be done) the user is to take to get to the correct bin orpart storage location, based at least in part on an analysis of sensordata regarding the location and/or orientation of the user (andassociated ARD 102) and the location of the correct bin or part storagelocation, wherein the sensor data can be received from one or moresensors (e.g., location sensor or GPS sensor) of the sensor component104. The IMC 108 can facilitate presenting, via the interface component106 (e.g., display component 302, audio component 304), the travel route(e.g., textual or audio travel directions) to the user and can guide theuser to the correct bin or part storage location to retrieve the partand/or an indicator (e.g., visual indicator and/or audio indicator) thatcan indicate the correct bin in which the part can be found. Forexample, with regard to a set of bins 406 in which parts are stored, theIMC 108 can facilitate presenting, via the interface component 106, thetravel route 408 to the correct bin 410 and an indicator 412 that canindicate the correct bin 410 and differentiate it from the other bins ofthe set of bins 406. The travel route 408 presented (e.g., displayed) onthe display screen to the user also can include a user locationindicator 414 that can indicate the location of the user along or inrelation to the travel route 408, and in relation to the correct bin410.

In response to selection of the part (e.g., from the correct bin or partstorage location) by the user, the IMC 108, employing the sensorcomponent 104, can determine whether the user selected the correct part.The sensor component 104 (e.g., image capture component 202) of the ARD102 can detect the part selected (e.g., picked up and held) by the userand can provide sensor information relating to detection of the partselected by the user to the IMC 108. The IMC 108 can determine the partthat was selected by the user based at least in part on the sensorinformation and part identification information obtained from the datastore 114, wherein the part identification information can facilitateidentifying the selected part. The IMC 108 can employ object recognitionto recognize or identify the part and/or can interpret a bar code orother identification information (e.g., symbol) associated with (e.g.,attached to or imprinted on) the part.

For example, the IMC 108 can compare an image of the part captured bythe sensor component 104 to images of parts stored in the data store 114and can determine an image of a part in the data store 114 that matchesor at least substantially matches the image of the part captured by thesensor component 104, wherein the image of the part in the data store114 can be associated with identification information that can identifythe part and can identify where and when the part is to be installed orotherwise assembled in the assembly process. Additionally oralternatively, the sensor component 104 (e.g., image capture component202 and/or scanner component 210) can capture an image of a bar code orother identification information associated with the part and/or canscan (e.g., read) the bar code or other identification informationassociated with the part. The sensor component 104 can communicate thesensor data relating to the bar code or other identification informationto the IMC 108. The IMC 108 can identify the part selected by the userbased at least in part on the results of analyzing such sensor data.

If the IMC 108 determines that the user selected the correct part, theIMC 108 can generate and facilitate presenting, via the interfacecomponent 106 (e.g., a display screen of the display component 302and/or an audio speaker of the audio component 304), a confirmationindicator or message. The confirmation indicator or message, forexample, can be a colored (e.g., green) indicator 416 that can emphasize(e.g., highlight) the selected part (e.g., the selected part can beoutlined in green in the visual display of the ARD 102) to indicate thecorrect part was selected and/or audio information (e.g., verbalmessage) confirming that the correct part was selected can becommunicated via the speaker to the user.

If, however, the user had selected the wrong part 418, and the dataanalysis by the IMC 108 caused the IMC 108 to determine that the userdid not select the correct part, the IMC 108 can generate and facilitatepresenting, via the interface component 106 (e.g., the display component302 and/or audio component 304), an incorrect part indicator or message.The incorrect part message, for example, can be a colored (e.g., red)indicator 420 that can emphasize the selected part (e.g., the selectedpart can be outlined in red on the display screen of the displaycomponent 302) to indicate the wrong part 418 was selected by the userand/or audio information indicating that the wrong part 418 was selectedcan be communicated via the speaker of the audio component 304 to theuser.

In response to determining the part 402 to be assembled or installed aspart of the task, the IMC 108 also can determine the location 422 andorientation on the product where the part 402 is to be assembled orinstalled and/or location and orientation of one or more other parts(e.g., 424, 426) or features of the product (e.g., in proximity to thelocation 422 where the part 402 is to be assembled or installed or towhich the part 402 is to be connected), based at least in part on theresults of the analysis of the assembly information (e.g., images of theproduct assembly and parts) relating to assembly of the product and/oranalysis of the part 402 and product. For instance, the ARD 102, via theIMC 108 and interface component 106 (e.g., through images or indicatorspresented on the display screen of display component 302 and/or audioinformation presented via the audio component 304), can show or indicateto the user where a part, component, or wire, etc., is to be placed(e.g., installed, connected, inserted, . . . ) inside a cabinet, device,unit, product, etc., associated with an assembly process, using thetechniques disclosed herein.

In some implementations, the IMC 108 also can determine the location ofthe user and/or the user's hands or tools (e.g., tool in hand) relativeto the product or part 402, and/or the location and orientation of thepart 402 (e.g., in the user's hand or on tool), based at least in parton sensor data obtained from the sensor component 104 relating tolocation of the user (and associated ARD 102). The IMC 108 canfacilitate emphasizing, via the display screen of the display component302 (and associated lens component) of the ARD 102, the location 422 onthe product where the part 402 is to be assembled based at least in parton such analysis results, wherein such analysis results can indicate thelocation 422 where the part 402 is to be assembled (e.g., relative tothe one or more other parts (e.g., 424, 426) or features of the productin proximity to the location where the part 402 is to be assembled or towhich the part 402 is to be connected). As part of the data analysis,the IMC 108 also can employ object recognition to facilitate identifyingthe location 422 of the installation or other assembling of the part 402on the product and/or identifying the one or more other parts (e.g.,424, 426) or features of the product in proximity to the location 422where the part 402 is to be installed or otherwise assembled or to whichthe part is to be connected, based at least in part on the assemblyinformation (e.g., images and/or textual information regarding therespective parts and features of the product).

The emphasis provided on the display screen of the display component 302can be, for example, an emphasis or highlight, such as a colored (e.g.,green) and shaped (e.g., circular or rectangular shaped) indicator 428around (e.g., surrounding) the location on the product where the part402 is to be assembled, an arrow indicator pointing at the location 422on the product where the part 402 is to be assembled, and/or a messagein proximity to the location 422 on the product where the part 402 is tobe assembled. For instance, the IMC 108 can determine the position inthe field of view on the display screen of the display component 302(and associated lens component of the ARD 102) that corresponds to(e.g., aligns with) the location 422 where the part 402 is to beinstalled or otherwise assembled on the product. The IMC 108 canfacilitate emphasizing, via the interface component 106, that positionon the display screen of the ARD 102 to facilitate emphasizing thecorresponding location 422 where the part 402 is to be assembled on theproduct. If the position changes due, for example, to the user movingthe ARD 102 (e.g., the user moving his head and the ARD 102 thereon),the IMC 108 can facilitate, via the interface component 106, moving theposition of the emphasis indicator 428 associated with the location 422on the display screen to continue to have the position on the displayscreen, and, accordingly, the emphasis indicator 428 on the displayscreen of the ARD 102, correspond to the location 422 where the part 402is to be assembled on the product.

In certain embodiments, in connection with the user installing,connecting, or otherwise assembling parts (e.g., components, wires, . .. ) on a product during assembly, the IMC 108 can generate customizedinformation (e.g., augmented and/or virtual reality information) thatcan be presented as a display presentation (e.g., virtual display of theproduct or portion thereof) on the display screen of the displaycomponent 302. With regard to parts that are not assembled into theproduct, the display presentation can show such parts in a washed-out(e.g., grayed-out or de-emphasized) manner With regard to respectiveparts of the product, as a part being assembled into the product, todepict the part as in the process of being assembled, the IMC 108 canfacilitate modifying the display presentation to display a virtual part,corresponding to that part, such that it is active or blinking (e.g.,intermittently transitioning between showing as washed-out or colored)in the display presentation. With regard to respective parts of theproduct, once the part has been assembled into the unit, to depict thepart as being assembled in the product, the IMC 108 can facilitatemodifying the display presentation to display the virtual part,corresponding to that part, such that it is solid or real (e.g., a solidcolor) in the display presentation.

In some implementations, to facilitate proper assembling of parts in aproduct, the IMC 108 can generate a display presentation (e.g., virtualdisplay presentation (e.g., virtual product) that can virtuallyrepresent the product) that can represent the product, or a portionthereof, based at least in part on the assembly information and/or otherinformation obtained from another data source(s). The displaypresentation can comprise a CAD model or wireframe model (e.g., 3-D CADor wireframe model), for example. The IMC 108 can facilitate presenting,via the display screen of the display component 302, the displaypresentation to the user. The user, using, for instance, hand gestures(e.g., captured by the image capture component 202), voice commands(e.g., captured by the audio sensor component 204), or controls of theARD 102 (e.g., received via the key/control component), can manipulatethe display presentation to zoom in to or out from a particular area ofthe virtual product (e.g., in the virtual view of the product on thevirtual display) and/or can rotate or otherwise adjust the viewingperspective of the virtual product (e.g., in the virtual view of theproduct on the virtual display). These disclosed aspects, along with theindicators relating to assembling of the part in the product (e.g.,indicators that can indicate the proper part and proper location toassemble the part in the product), can enhance assembling of parts inthe product.

If, for some reason, the user is attempting to install or otherwiseassemble the part 402 in the wrong place 430 on the product, the IMC 108can identify that the user is attempting to assemble the part 402 in thewrong place 430 on the product, based at least in part on the analysisresults relating to determining the correct location 422 on the productto assemble the part 402, as disclosed herein. The IMC 108 canfacilitate, via the interface component 106, presenting a negativeindicator 432 on the display screen of the ARD 102, wherein the negativeindicator 432 can indicate to the user that the user is attempting toassemble the part 402 in the wrong location 430 on the product. Thenegative indicator 432 can comprise, for example, a colored (e.g., red)indicator of desired form (e.g., a red circle with a line through it)that can be placed over a location on the display screen that cancorrespond to the location 430 (e.g., wrong location) on the productwhere the user is improperly attempting to assemble the part 402 in theproduct. Additionally or alternatively, the IMC 108 can present (e.g.,via a speaker of the audio component 304 and/or the haptic component306) an indicator (e.g., verbal message and/or haptic communication) tothe user to notify the user that the user is attempting to assemble thepart 402 in the wrong place 430 on the product.

In a similar regard, the ARD 102 also can be utilized to determinewhether the user of the ARD 102 is in the right place or wrong place toperform work tasks and/or is assembling the right unit or wrong unit.For instance, the IMC 108 can determine the location and orientation ofthe user based at least in part on sensor data relating to the user'slocation and orientation received from one or more sensors of the sensorcomponent 104, as more fully disclosed herein. The IMC 108 also candetermine the correct location where the user should be at to performthe next work task and/or the correct unit (e.g., product, cabinet,device, . . . ) on which the user is to perform the next work task,based at least in part on the results of analyzing sensor data relatingto the user and/or unit received from the sensors of the sensorcomponent 104, assembly information and/or work instructions relating tothe assembly of the unit, and/or other information.

If the IMC 108 determines that the user is in the wrong location (e.g.,in a building or other area), is attempting to perform a work task onthe wrong unit, and/or is not facing the correct unit (e.g., accordingto the orientation of the user) to perform the next work task, the IMC108 can communicate one or more indicators (e.g., visual indicator,audio indicator, and/or haptic indicator) via the interface component106 (e.g., display screen of display component 302, audio speaker ofaudio component 304, and/or haptic emitter of the haptic component 306)to the user to notify the user that the user is in the wrong location,is attempting to perform the work task on the wrong unit, and/or is notfacing the correct unit to perform the next work task. The IMC 108 alsocan determine and generate a travel route to guide the user from thecurrent (e.g., wrong) location to the correct location where the correctunit is located, based at least in part on the current location of theuser and the location of the unit, using the techniques disclosedherein. The information can facilitate presenting, via the interfacecomponent 106 (e.g., via display screen of the display component 302and/or speaker of the audio component 304), the travel route to theuser. The travel route generated by the IMC 108 in such instances when auser is in the wrong location to perform work tasks can be particularlyuseful, for example, in larger areas or buildings.

In a manner similar to the above disclosed subject matter regarding theassembling of the correct part in or on a product, the ARD 102 also canbe employed to confirm that a part is assembled (e.g., installed orconnected) correctly and/or determine whether a part is connectedcorrectly to other parts of a product, and can provide (e.g.,communicate) confirmation that the part is connected correctly orprovide a negative indicator to indicate that the part is not connectedcorrectly and/or has somehow become incorrectly connected. For instance,the ARD 102, employing the sensor component 104 and IMC 108, can confirmthat appropriate nodes (e.g., screw terminals, wire ends, tabs,brackets, . . . ) of a product are in contact with the appropriate nodesof surrounding and associated parts of the product.

For example, the sensor component 104 can sense various connectionsbetween parts of a product, for example, by capturing images of thevarious connections of the product (e.g., via the image capturecomponent 202), sensing data flow between respective components of theproduct (e.g. via one or more other sensors 216), and/or other sensingmeans, and can generate sensor data relating thereto. The IMC 108 cananalyze all or a portion of the sensor data obtained from the sensorcomponent 104 and/or other data relating to the product from anotherdata source(s) to facilitate determining whether parts on the productare appropriately connected to each other. For instance, the IMC 108 cancompare the images of the various connections between parts of theproduct (e.g., images received from the sensor component 104) to storedimages regarding the correct connections between parts of the product,and can determine whether the various connections between the parts ofthe product are correct based at least in part on the results of thecomparison. The IMC 108 can retrieve the stored images from the datastore 114 or an external data source (e.g., via the communicationnetwork).

If the IMC 108 determines that the connections between the parts of theproduct are correct, based at least in part on the analysis results, theIMC 108 can facilitate presenting, via the interface component 106(e.g., the display screen of the display component 302 and/or a speakerof the audio component 304), a positive indicator (e.g., a green orotherwise confirming visual indicator, or a confirmation audio message).If the IMC 108 determines that a connection between parts of the productis not correct or is otherwise faulty, based at least in part on theanalysis results, the IMC 108 can facilitate presenting, via theinterface component 106 (e.g., the display screen of the displaycomponent 302 and/or a speaker of the audio component 304), a negativeindicator (e.g., a red visual indicator or other bad-connection visualindicator, or a bad-connection audio message) to the user to indicatethat such connection between those parts of the product is not corrector is otherwise faulty.

In some instances, there can be an issue with regard to tracing a partof a product (e.g., tracing or identifying a serial number of the part).The ARD 102 can be employed to facilitate tracing or identifying thepart. For instance, the sensor component 104 can employ the imagecapture component 202 to capture an image of a part code of a partand/or capture images of the part, the scanner component 210 to scan thepart code of the part, and/or another sensor(s) of the sensor component104 to sense other information regarding the part that can be used tofacilitate tracing the part, wherein the part code can be a serialnumber, bar code, two-dimensional (2-D) code, or other type of partcode. In some implementations, the image capture component 202 oranother sensor(s) can capture or scan the part code, or at least captureimages of the part, in response to sensing that the user is looking atthe part through the lens component of the ARD 102. Such other sensor(s)of the sensor component 104 can comprise, for example, sensors that canperform or employ near field imaging (NFI), Bluetooth technology, radiofrequency (RF) technology, infrared (IR) technology, and/or othertechnology to sense or capture information regarding a part that can beused to facilitate tracing or identifying the part. The respectivesensors of the sensor component 104 can generate sensor data based atleast in part on the respective sensing performed by the respectivesensors. The IMC 108 also can request that the user otherwise confirmthe identification of the part.

The tracing, identifying, and/or confirming of parts of the product,using the disclosed techniques, can facilitate enhancing the assemblyprocess, including enhancing bill of materials (BOM) confirmation and aserial numbers-in-this-product list). The ARD 102, employing the IMC108, also can facilitate verification of the assembly (e.g., properassembly) of the product, based at least in part on the tracing,identifying, and/or confirming of the parts of the product, andcomparing the 3-D view of the display presentation of the virtualproduct to the BOM associated with the product, to the print associatedwith the product, to the view of the camera(s) of, or images of theproduct captured by, the image capture component 202, and/or to the liveview presented via the lens component of the ARD 102.

For example, the IMC 108 and the sensor component 104 (e.g., imagecapture component 202 or other sensor(s)) of the ARD 102 can check(e.g., double check or confirm) measurements, for example, of wires,hole placements, hole sizes, components (e.g., component sizes and/orrelative placement of components to other components), etc., of aproduct. The image capture component 202 can capture images of thewires, hole placements, hole sizes, component sizes, relative placementof components to other components, etc., of the product. The IMC 108 canretrieve stored images relating to the measurements associated with theproduct from the data store 114 or an external data source. The IMC 108can compare the captured images to the stored images to determinewhether there are any unacceptable differences between the measurementsidentified in the captured images and the measurements identified in thestored images, based at least in part on defined assembly criteriaassociated with the product.

In response to determining that the measurements identified in thecaptured images satisfy the defined assembly criteria, the IMC 108 canfacilitate presenting, via the interface component 106, a confirmationmessage or indicator (e.g., visual or audio message or indicator) thatcan confirm that the assembled product (or portion thereof underinspection) satisfies the defined assembly criteria. In response todetermining that the measurements identified in the captured images donot satisfy the defined assembly criteria (e.g., because one or moremeasurements in the captured images do not suitably compare to one ormore corresponding measurements in the stored images), the IMC 108 canfacilitate presenting, via the interface component 106, an assemblyproblem message or indicator (e.g., visual or audio message orindicator) that can indicate that the assembled product (or portionthereof under inspection) does not satisfy the defined assembly criteriaand/or can indicate which portion of the assembly does not satisfy thedefined assembly criteria.

In some implementations, the IMC 108 can receive the sensor data fromthe sensor component 104, and can access a database (e.g., externaldatabase) that can comprise part identification information (e.g., partcodes, serial numbers of parts, bar codes of parts, images of parts, 2-Dcodes, . . . ) that can identify various parts. The IMC 108 can comparethe sensor data to the part identification information accessed from thedatabase to generate comparison (e.g., analysis) results. The IMC 108can determine the part code based at least in part on the analysis(e.g., comparison) results.

In other implementations, the IMC 108 can receive the sensor datarelating to the part from the sensor component 104, and can upload thesensor data relating to the part (e.g., sensor data relating toidentification of the part) to another system (not shown in FIG. 1),such as a manufacturing execution system or part identification system.Such other system can trace or identify the part, and can generate partidentification information that can identify the part, based at least inpart on the sensor data received from the IMC 108. The IMC 108 canreceive the part identification information from such other system andcan use the part identification information to identify the part, and/orto make other decisions and/or determinations relating to the part.

The ARD 102 also can enable identification of a part that does not meetinspection criteria (e.g., incoming inspection criteria). For instance,with regard to a part, the IMC 108 can identify the part, using thedisclosed techniques, and can obtain a photograph(s) or visual view(e.g., 2-D or 3-D visual view) of the part from the data store 114 or anexternal data source, based at least in part on the part identification.The visual view of the part can be a CAD drawing of the part and/or amanipulative visual view (e.g., a 3-D visual view that the user canmanipulate using controls or commands via the ARD 102 to adjust the viewof the part with respect to the user), for example. The IMC 108 canfacilitate, presenting via the display screen of the interface component106, the photograph(s) or visual view of the part to the user. The usercan compare the photograph(s) or visual view of the part to the liveview of the part as the user views the part through the lens componentof the ARD 102. The user can determine whether the part satisfies thequality criteria for the part based at least in part on the results ofcomparing the photograph(s) or visual view of the part to the actualpart.

Additionally or alternatively, the user can use the image capturecomponent 202 (e.g., one or more cameras) of the interface component 106to capture one or more images of the part, wherein the one or moreimages can capture dimensions and/or other quality-related features(e.g., critical dimensions and/or critical-to-quality aspects) of thepart. The IMC 108 and/or the user can compare the dimensions and/orother quality-related features of the part to information (e.g., images,specifications, quality criteria, . . . ) relating to the dimensionsand/or other quality-related features of the part, and/or qualitycriteria applicable to the part, wherein the IMC 108 can obtain suchinformation from the data store 114 or an external data source (e.g., acompany database). Based at least in part on the results of comparingthe dimensions and/or other quality-related features of the part to theinformation relating to the dimensions of the part, otherquality-related features of the part, and/or quality criteria applicableto the part, the IMC 108 or user can determine whether the partsatisfies the quality criteria applicable to the part.

In some implementations, the ARD 102 can be employed to identifyinstances when a part(s) from an assembly of a product is missing or hasbeen forgotten. For instance, the image capture component 202 of the ARD102 can capture one or more images of a view of the product assembly.The IMC 108 can obtain one or more assembly views (e.g., expectedassembly views) of the product assembly from the data store 114 or anexternal data source (e.g., company database) and/or assemblyinformation (e.g., assembly instructions) relating to the productassembly. The IMC 108 can analyze the one or more images of the view ofthe product assembly and the one or more assembly views and/or theassembly information. For instance, the IMC 108 can compare the one ormore images of the view of the product assembly to the one or moreassembly views and/or the assembly information to facilitate determiningwhether a part of the product assembly has been forgotten or isotherwise missing.

Based at least in part on the results of the analysis, the IMC 108 candetermine whether a part of the product assembly has been forgotten oris otherwise missing. If the IMC 108 determines that a part of theproduct assembly is missing, the IMC 108 can generate an assembly errormessage, which can include a notification that the part is missing fromthe product assembly and/or other information regarding the missingpart. The interface component 106 can communicate the assembly errormessage to the user via the display screen of the display component 302,a speaker of the audio component 304, and/or a haptic feedback signalfrom the haptic component 306 to notify the user that the part ismissing from the product assembly and/or the other information regardingthe missing part (e.g., identify which part is missing and/or thelocation on the product assembly where the part is supposed to beinstalled).

The ARD 102 also can be employed to train users to perform works tasksfor product assembly and/or can enable untrained or undertrained usersto perform the work tasks for product assembly (e.g., by learning andbeing trained on the job), even though they are untrained orundertrained. Training materials often can be more efficacious when theyinvolve pictures as, for example, pictures do not need languagetranslation and pictures can be closer to reality (e.g., visual reality)than verbal or written descriptions.

The ARD 102 can be used to develop, generate, and provide workinstructions by capturing a desired number of images (e.g., photographs)from various perspectives and desired (e.g., expert) user instructionsfrom a desired article build (e.g., a first-article build) of theproduct. As the product is being assembled during the desired articlebuild, the trained (e.g., expert) user of the ARD 102 can capture imagesof different portions of the assembly process for the product (e.g., toprovide work instructions on a part-by-part, step-by-step basis). Theimages captured by the ARD 102 can be still frames (e.g., still images)or video of what the camera(s) of the image capture component 202perceives or captures during the assembly process of the desired articlebuild of the product. The trained user (e.g., teaching assembler) alsocan use the audio sensor component 204 (e.g., microphone) to provide(e.g., communicate) verbal work instructions regarding performing thework tasks of the assembly process (e.g., describing what the traineduser is doing) and/or the key/control component 308 to provide textualwork instructions regarding performing the work tasks of the assemblyprocess. As desired, the IMC 108 and/or another component of the ARD 102can employ a speech-to-text engine to convert speech (e.g., verbal workinstructions) of the trained user to textual information (e.g., textualwork instructions), which can be edited, if necessary, and included inthe work instructions.

The ARD 102 also can obtain images and/or other information regardingthe product assembly from another data source(s) (e.g., an externaldatabase(s)) or other persons working on the product assembly. Asdesired (e.g., as appropriate), the ARD 102, employing the IMC 108, alsocan create or obtain CAD models (e.g., 3-D and/or virtual CAD models) ofthe product, the product assembly, or respective portions thereof, thatcan be used as part of the work instructions. The CAD models can be usedto augment the images of the product assembly. In some implementations,a CAD model can be manipulated by the user, using various controls ofthe interface component 106, to view the CAD model from variousperspectives, via the display screen of the display component 302 of theARD 102. For example, the IMC 108 can control manipulation of the CADmodel, and can adjust the view of the CAD model with respect to theuser, based at least in part on hand gestures (e.g., captured via theimage capture component 202), voice commands (e.g., captured by theaudio sensor component 204), and/or textual commands (e.g., received viathe key/control component 308) presented by the user.

The work instructions, comprising the images, verbal work instructions,and/or the textual work instructions, can be available to other userswho can use their ARDs 102 to access the work instructions from the datastore 114 of the ARD 102 or from an external data source. A user (e.g.,an untrained or undertrained user) of an ARD 102 can play back the workinstructions on the ARD 102 (e.g., on the display screen and through theaudio speaker on the ARD 102) so the user can be trained with regard toperforming the work tasks of the assembly process and/or so that theuser can perform the work tasks of the assembly process (e.g., as theuser learns on the job). In some implementations, the ARD 102 can uselive 2-D or 3-D video, audio (e.g., audio instructions), and/or hapticfeedback, presented to the user via the interface component 106, forfully immersive guidance (e.g., live guidance) of the user or training(e.g., beforehand) of the user regarding the assembly procedure of theproduct.

In some embodiments, a user of the ARD 102 can use the image capturecomponent 202 and/or other components of the ARD 102 to capture imagesand/or other information (e.g., audio signals) of a view(s) of anassembly of a product, and can communicate (e.g., broadcast or transmit)the images and/or other information of the view(s) of the productassembly to another device (e.g., another ARD, or a communicationdevice, such as a computer, electronic tablet, or mobile phone) ofanother user (e.g., supervisor, trained user, and/or expert user) forreview of the assembly work by the other user and/or live guidanceregarding the assembly (e.g., how to perform an assembly task) from theother user. The user can receive, via the ARD 102, feedback (e.g., videoand/or audio communications or information) from the other user, via theother device, to indicate whether the user has performed the assemblycorrectly or provide the live guidance regarding the assembly to theuser.

As disclosed, the ARD 102 can be employed to create and use workinstructions regarding a product assembly, wherein the work instructionscan be step-by-step work instructions. By presenting step-by-step workinstructions to the user via the ARD 102 (e.g., via the display screenof the display component 302 and/or speaker of the audio component 304),it can enable the ARD 102 to lead, instruct, or train the user toperform the work tasks regarding the product assembly in relativelysmall bits (e.g., installation or construction part by part) and reviewand check the work of the user (e.g., after each portion (e.g.,performance of each work task) of the product assembly process), whichcan effectively enable virtually untrained users to perform the worktasks.

In certain embodiments, the ARD 102 can be utilized to secure the ARD102, other devices or information (e.g., external databases) associatedwith the ARD 102, and/or an area(s) associated with the ARD 102 frombeing accessed by unauthorized persons. The ARD 102 can verify orfacilitate verifying the identity of a user based at least in part onauthentication information (e.g., authentication credential) receivedfrom the user. The IMC 108 can control access to the ARD 102 based atleast in part on whether the user is an authorized user. The IMC 108 canrequest that the user provide (e.g., present or submit) authenticationinformation to the ARD 102.

The authentication information can be in the form of a passcode (e.g.,password, pass phrase, personal identification number (PIN), . . . ),gesture(s) (e.g., one or more finger and/or hand gestures of the user inthe form of a finger or hand passcode), biometric information of theuser, and/or other types of authentication information. A textualpasscode can be received via the key/control component 308 of the ARD102 from the user, wherein the user can use the keys or controls toenter the passcode. A verbal passcode can be received via the audiosensor component 204 (e.g., microphone) from the user, wherein the usercan speak the passcode into the audio sensor component 204. Thebiometric information can comprise, for example, fingerprint, hand printor hand structure, eye (e.g., iris or retina) features, facial features(e.g., facial dimensions), and/or voice features, of the user. The ARD102 can employ the user-facing camera(s) of the image capture component202 to obtain (e.g., capture an image(s) of) the biometric informationrelating to the eyes or face, and/or can use the outward-facingcamera(s) of the image capture component 202 to capture the biometricinformation relating to the fingers or hand. The ARD 102 can utilize theaudio sensor component 204 to obtain (e.g., capture) the voice featuresof the user, wherein the user can speak into the audio sensor component204 to enable the ARD 102 to obtain voice information of the user.

The IMC 108 can analyze the received authentication information andstored authentication information. For instance, the IMC 108 can comparethe received authentication information to the stored authenticationinformation to determine whether the received authentication informationmatches, or at least substantially (e.g., sufficiently) matches, thestored authentication information. If the IMC 108 determines that thereceived authentication information matches or at least substantiallymatches the stored authentication information, the IMC 108 can determinethat the user is authenticated and can grant the user a set of accessrights to the ARD 102, associated devices and/or information, and/or thearea(s) associated with the ARD 102, in accordance with the accessrights the user is authorized to have. If the IMC 108 determines thatthe received authentication information does not match or substantiallymatch the stored authentication information, the IMC 108 can determinethat the user is not authenticated and can deny the user access rights,or can grant the user only limited access rights, to the ARD 102,associated devices and/or information, and/or the area(s) associatedwith the ARD 102, wherein the limited access rights can be, for example,a subset of access rights that can be granted to an unauthenticateduser.

In other embodiments, the ARD 102 can communicate the receivedauthentication information of the user to another device (e.g., a devicethat can perform authentication or security authorizations and isassociated with a company database comprising security andauthentication information), wherein the other device can perform suchanalysis (e.g., comparison) of the received authentication informationand the stored authentication information. The other device cancommunicate with the ARD 102 to inform the ARD 102 with regard towhether the user has been authenticated or not, and/or to inform the ARD102 of the particular access rights that can be granted to the user.

In a manner similar to product assembly activities and tasks, the ARD102 also can be used for enhancing maintenance activities (e.g.,maintenance assistance), for example, associated with an assemblyprocess, an industrial automation system, or with regard to another typeof environment. Certain problems may arise during maintenance. As someexamples, there may be hazards inside an electrical cabinet or otherdevice to be maintained which are obscured by the cabinet door in normalcircumstances. Before performing maintenance work safely, circuits oftenshould be (e.g., need to be) shut down. If the controls to shut thecircuits down are remote or within the cabinet itself, this can causedifficulties and/or present potential hazards. Parts sometimes can wearout after they are used/stressed in certain ways. While this data may beavailable inside a controller or a database, it can be difficult toobtain and may in fact not even be considered except when a given partfails. Determining which cabinet in a plant and which part in a systemcontains a serviceable part can be prone to mistakes, particularly whenthere are many similar cabinets or many similar parts. There also can beproblems which are associated with the reassembly portion ofmaintenance.

It is to be appreciated and understood that aspects, embodiments, andtechniques disclosed herein with regard to assembly activities or tasksperformed or facilitated (e.g., enabled, enhanced, . . . ) by the ARD102 generally can be applicable to maintenance activities or tasksperformed or facilitated by the ARD 102. Likewise, it is to beappreciated and understood that aspects, embodiments, and techniquesdisclosed herein with regard to maintenance activities or tasksperformed or facilitated by the ARD 102 generally can be applicable toassembly activities or tasks performed or facilitated by the ARD 102.

In certain implementations, with the user (e.g., operator) interactingwith or wearing the ARD 102, the ARD 102 can assist the user in a numberof ways, such as disclosed herein, to improve speed of the user withregard to performing maintenance tasks, quality of maintenanceperformed, security of the performance of maintenance tasks, and safetyof the user and the environment in the performance of maintenance tasks.The ARD 102, employing the IMC 108 and the interface component 106, canadvise or instruct the user through visual signals, audible signals,and/or haptic feedback to facilitate enhancing and assisting in theperformance of maintenance tasks associated with the system (e.g.,enhancing the speed, quality, security, and safety associated with theperformance of maintenance tasks associated with the system).

One issue that can arise in maintenance of a unit, such as an electricalcabinet or other device, is that there may be hazards inside the unitthat, under normal circumstances, can be obscured from the user by acabinet door of the unit. The ARD 102, employing the IMC 108 and theinterface component 106, can generate and present visible, audible,and/or haptic alerts, notifications, or warnings to the user before theuser opens the unit (e.g., alert, notification, or warning regardinghigh voltage, high temperature, etc.; and/or actual or virtual warninglabels from the equipment designers). The ARD 102 can provide theinformation to the user textually on the display screen of the displaycomponent 302 (e.g., as if written on the door of the unit), audibly viaan audio message presented by the audio component 304, haptically byemitting haptic feedback from the haptic component 306 when the userapproaches any hazards, and/or visually on the display screen of thedisplay component 302 (e.g., as if the user is looking through the doorinto the unit and showing internal components and associated warnings,voltages, temperatures, in textual or virtual meters or a “heat-map” ofhazards).

When a user of the ARD 102 is in proximity to the unit to perform amaintenance task or when the IMC 108 determines that the user is toperform, or is preparing to perform, a maintenance task on the unit, oneor more sensors of the sensor component 104 can sense the locationand/or orientation of the user, and can sense conditions associated withthe unit (e.g., inside or outside the unit) and/or the area around theunit, and can generate sensor data relating thereto, for example, usingthe techniques disclosed herein. The IMC 108 can determine the locationand/or orientation of the user, including the location and/orientationof the user relative to the unit, based at least in part on the sensordata.

The IMC 108 also can determine one or more hazards associated with theunit based at least in part on the sensor data relating to theconditions associated with the unit and/or other data obtained from thedata store 114 or another data source(s). For instance, the camera(s) ofthe image capture component 202 can comprise a temperature sensor(s)(e.g., IR sensor(s)) that can sense or perceive temperatures (e.g., IRtemperatures) inside or outside of the unit. The IMC 108 can determineand generate (e.g., automatically and/or virtually instantaneouslydetermine and generate) customized information (e.g., augmented and/orvirtual reality information), comprising a heat map of respectivetemperatures of respective regions or components inside or outside theunit, based at least in part on the sensor data relating to thetemperatures inside or outside of the unit. For example, the IMC 108 cantranslate (e.g., automatically and virtually instantaneously translate)the IR temperatures into corresponding colorations (e.g., differentcolors representing different temperature levels) in the form of a heatmap that can be overlaid on the view on the display screen (e.g., of thedisplay component 302) through the glasses (e.g., lens component) of theARD 102 to enable the user to avoid touching hazardous (e.g., hot)components of the unit. The IMC 108 also can obtain data regardingcomponents of the unit that may be hazardous (e.g., hot) from the otherdata (e.g., specifications or descriptions of components or the unit)obtained from the data store 114 or another data source(s) and canincorporate such other data into the heat map.

Referring to FIG. 5 (along with FIG. 1), FIG. 5 illustrates a diagram ofvarious example visual presentations 500 in the field of view of a userin connection with performing maintenance tasks or other tasks, inaccordance with various aspects and embodiments of the disclosed subjectmatter. The visual presentations 500 depicted in FIG. 5 can include avisual presentation 502 of a heat map 504 illustrating respective heatlevels inside or in proximity to the unit 506. The IMC 108 can generatethe heat map 504 to have a desired granularity with regard to thedisplaying the respective temperatures of respective areas inside or inproximity to the unit 506, wherein different colors or colorations canrepresent different temperature levels (e.g., areas with darker redcoloring can indicate hotter temperatures, areas with lighter colors(or, alternatively, darker blue colors) can indicate coolertemperatures). For example, in response to determining that at least aportion (e.g., internal component) of the inside of the unit 506 has ahigher (e.g., hot and potentially dangerous temperature), the IMC 108can use a first visual representation or indicator 508 that can have afirst color (e.g., dark red) on the heat map 504 to indicate thatportion of the inside of the unit has a higher temperature. Based atleast in part on the IMC 108 determining that another portion of theinside of the unit has a relatively lower temperature, the IMC 108 canuse a second visual representation or indicator 510 that can have asecond color (e.g., blue) on the heat map 504 to indicate that otherportion of the inside of the unit has a relatively lower temperature.Other portions of the heat map 504 can be respectively colored usingrespective visual representations or indicators according to theirrespective temperature levels.

In some implementations, based at least in part on the results ofanalyzing the sensor data or the other data, the IMC 108 also candetermine one or more other hazards, such as high voltages or highcurrents, that can be inside or associated with the unit 506. Inresponse to determining one or more hazards in or associated with theunit, the IMC 108 can generate one or more visual presentations (e.g.,indicators, virtual meters, . . . ) and can facilitate presenting, viathe interface component 106, the one or more visual presentations to theuser (e.g., by overlaying the display presentation(s) on the displayscreen associated with the lens component of the ARD 102). For instance,the IMC 108 can generate a visual indicator that can indicate anelectrical hazard (e.g., high voltage, high current) in or associatedwith the unit (e.g., a component inside the unit) and/or can present avirtual meter(s) illustrating a voltage level or current level in orassociated with the unit (e.g., a component inside the unit).

For instance, as illustrated in the visual presentations 500 of FIG. 5,the IMC 108 can generate and facilitate presenting (e.g., overlaying),via the display screen of the interface component 106 associated withthe lens component of the ARD 102, a visual presentation 520 that caninclude one or more objects 522 (e.g., virtual objects or indicators)that can represent and indicate the hazardous or at least potentiallyhazardous conditions inside or associated with the unit 506. The one ormore objects 522 can comprise, for example, a high voltage symbol 524,textual information 526 (e.g., HIGH VOLTAGE (HV)), or a virtual meter528 (e.g., a virtual voltage meter (as depicted), or virtual currentmeter). With regard to temperatures, in addition to or as an alternativeto the heat map 504, the one or more objects 522 can include a hazardoustemperature indicator 530 that can indicate a hazardous (e.g., hot) orpotentially hazardous temperature has been identified in the region(e.g., inside the unit 506) at or near where the hazardous temperatureindicator 530 is located in the visual presentation 520.

In some implementations, the IMC 108 can obtain images of actual warninglabels provided by the manufacturer of the unit, or component therein,or provided by another data source or entity. In addition or as analternative to the objects or indicators depicted in the visualpresentations (e.g., 502, 520), the IMC 108 can generate and facilitatepresenting, via a visual presentation on the display screen of the ARD102, a virtual warning label that can correspond to an actual warninglabel with respect to the unit 506 or component therein.

Additionally or alternatively, the IMC 108 can generate an audioindicator that can be communicated by the audio component 304 to notifyor warn the user of the hazard, the location of the hazard (e.g., insidethe unit, or a particular place inside the unit), a component associatedwith the hazard, the type of hazard, and/or parameter values (e.g.,voltage level, current level) associated with the hazard, etc.Additionally or alternatively, the IMC 108 can generate a hapticindicator that can be presented by the haptic component 306 to notify orwarn the user that the user is in proximity to a hazard, for example, inresponse to the IMC 108 determining that the user is in proximity to thehazard.

In some embodiments, the IMC 108 can determine and generate customizedinformation (e.g., augmented and/or virtual reality information),comprising a visual presentation 540 (e.g., a virtual view) ofcomponents, wires, connections, etc., of the unit 506 (e.g., inside orassociated with the unit 506) based at least in part on the sensor dataand/or the other data. The visual presentation 540 can comprise virtualobjects (e.g., virtual components, such as virtual components 542, 544,and 546; virtual wires, such as virtual wire 548; and/or other virtualconnections, such as virtual connection 550; . . . ) that illustrate orrepresent the interrelationships or associations between components(e.g., internal components of the unit), wires, connections, etc.,inside or associated with the unit. For instance, the IMC 108 can haveimages (e.g., photographs, IR images, . . . ) of a component of the unit506 that were captured by the image capture component 202 of the ARD 102(or by another ARD 102 or another device), can have other images of thecomponent obtained from another data source or entity (e.g., themanufacturer of the component), and/or can have other visualrepresentations (e.g., a 3-D CAD or wireframe model) of the component.The IMC 108 can determine and generate a virtual component (e.g., 542)of the component based at least in part on the results of analyzing thecaptured images of the component, other images of the component, and/orother visual representations of the component.

As desired (e.g., by the user), this visual presentation 540 can becombined with (e.g., integrated with) other visual presentations, such,for example, visual presentations 502 or 520 (e.g., visual indicator,virtual meter, heat map, . . . ), and/or other visual information (e.g.,other customized visual information), such as parameter data ortask-related information. For example, a heat map 504 can be overlaid onthe visual presentation 540 of the virtual objects, or a visualindicator (e.g., high voltage indicator, such as high voltage symbol524) can be overlaid on a virtual component (e.g., 542) in the visualpresentation 540 that corresponds to a component in the unit determinedby the IMC 108 to have a high and potentially dangerous voltage.

The ARD 102, by providing one or more of the visual presentations (e.g.,502, 520, and/or 540) can enable a user to see inside (e.g., virtuallysee inside) the unit (e.g., via the virtual unit) before opening theunit. This can facilitate enabling the user to make decisions regardingan internal component(s) of the unit before the user opens the unit oraccesses the inside of the unit, and/or enabling the user to takeprecautions (e.g., proper precautions) against hazards (e.g., highvoltage, hot temperatures, . . . ) that are or may be inside of theunit, using the feedback (e.g., feedback regarding voltage ortemperature) obtained by the sensor component 104 (e.g., IR sensor)and/or the other data obtained by the IMC 108, as such sensor dataand/or other data is analyzed and processed by the IMC 108.

The user also can use the ARD 102 to facilitate planning or rehearsingwork to be performed inside the unit (e.g., performed on an internalcomponent(s) inside the unit) before opening up the unit (e.g.,cabinet). For instance, prior to performing work inside the unit, theuser can use the ARD 102 to virtually interact with an internalcomponent(s) (e.g., interact with a virtual version (e.g., 542) of theinternal component(s) on the visual presentation 540) to facilitateplanning, rehearsing, and/or simulating work that is to be performedinside the unit 506 (e.g., performed on the internal component(s) insidethe unit 506). For example, the user can use hand gestures (e.g.,captured or detected by the image capture component 202), voice commands(e.g., input via the audio sensor component 204), and/or textual orcontrol commands (e.g., input via the key/control component 308, tomanipulate controls, sub-components, etc. (e.g., virtual controls,virtual sub-components), of the virtual component (e.g., 542) and/orconnections (e.g., virtual connections (e.g., 550)) associated with thevirtual component, and/or manipulate the user's view of the virtualcomponent or associated connections to facilitate planning, rehearsing,and/or simulating work that is to be performed on the component insidethe unit.

The user also can use the ARD 102 to confirm (e.g., by location of theARD 102 and user and/or by a virtual in-cabinet view of the unit) thatthat the user is at the correct place where work is to be performed bythe user and/or is at or interacting with the correct unit on which workis to be performed by the user. The IMC 108 can determine the locationand/or orientation of the user, based at least in part on the results ofanalyzing sensor data from one or more sensors of the sensor component104 and/or other data, as more fully described herein. The IMC 108 alsocan determine the location of the unit and the relative location of theuser to the unit, based at least in part on the results of analyzing thesensor data and/or other data, as more fully described herein. Based atleast in part on the respective locations of the user (and associatedARD 102) and the unit and/or orientation of the user, the IMC 108 candetermine whether the user is in the correct place (e.g., in front of orin proximity to the unit) where work is to be performed by the userand/or whether the user is interacting with the correct unit on whichwork is to be performed by the user.

In order to safely perform maintenance work involving electricalcircuits, before performing maintenance work, it often can be desirableto shut down circuits that are or may be associated with the maintenancework. If the controls to shut the circuits down are remote or within theunit itself, this may cause difficulties for the user attempting to shutdown the circuits and/or expose the user to hazards. The IMC 108 of theARD 102 can access system design parameters from the data store 114 oran external data source(s) (e.g., a database). The IMC 108 can determinewhich circuits are associated with (e.g., connected to) which breakersof the system based at least in part on the results of analyzing thesystem design parameters. Accordingly, the IMC 108 can generatecustomized information (e.g., augmented and/or virtual realityinformation) that can indicate or illustrate which circuits areassociated with (e.g., connected to) which breakers of the system. TheIMC 108 can facilitate presenting (e.g., overlaying), via the displayscreen of the interface component 106, the customized information to theuser. For example, in response to detecting the user in proximity to theequipment on which maintenance work is to be performed and/or when theIMC 108 identifies that the user is preparing to perform the maintenancework on the equipment, the IMC 108 can facilitate presenting, via thedisplay screen of the interface component 106, the customizedinformation to the user.

Further, the IMC 108 can determine or confirm the location of all thedevices, components, etc., associated with the maintenance work, basedat least in part on the results of analyzing sensor data and/or otherdata, using the techniques described herein. The IMC 108 can determineand facilitate presenting (e.g., overlaying) respective customizedinformation (e.g., respective pertinent information) associated with therespective devices, components, etc., virtually within the field of viewof the user (e.g., via the display screen of the interface component 106associated with the lens component of the ARD 102).

In some instances, these circuits that are to be shut down (e.g., turnedoff, or disconnected, . . . ) in connection with the maintenance workcan be controlled or shut down by a PLC or other logically controlleddevices. In such instances, the IMC 108 can generate a virtual controlpanel comprising virtual controls and/or other controls (e.g., switches,estop, . . . ) that can represent and virtually recreate the controls ofthe PLC or other logically controlled devices, based at least in part ondata relating to the PLC or other logically controlled devices that canbe obtained from the data store 114 or other data source (e.g., externaldatabase). The IMC 108 also can facilitate establishing a connection(e.g., a wireless communication connection) between the ARD 102 and thePLC or other logically controlled devices (e.g., via a communicationnetwork or direct communication connection). Using the ARD 102, the usercan make gesture commands, which can be captured by the image capturecomponent 202, to manipulate the virtual controls on the virtual controlpanel, or other controls or switches, to facilitate adjusting the actualcontrols, switches, etc., of the PLC or other logically controlleddevices. The IMC 108 can determine the manipulations or adjustments madeto the virtual controls on the virtual control panel, or other controlsor switches, by the user, based at least in part on the capturedgestures of the user (e.g., finger, hand, eye, and/or facial gestures ofthe user). The IMC 108 can adjust the virtual controls on the virtualcontrol panel, or other controls or switches and/or can generate controlinformation, based at least in part on the determined manipulations oradjustments. The IMC 108 can communicate the control information (e.g.,control instructions) to the PLC or other logically controlled devicesvia the connection to facilitate adjusting the actual controls,switches, etc., of the PLC or other logically controlled devices. Inresponse, the actual controls, switches, etc., of the PLC or otherlogically controlled devices can be adjusted accordingly, based at leastin part on the control information, to shut down the associatedcircuits. Such shut down of the circuits by the user can be performedusing the ARD 102 before certain safety processes, such aslock-out-tag-out, for example. For example, using the ARD 102, the usercan control the PLC or other logically controlled devices, or components(e.g., internal components of a cabinet) associated therewith, toperform a safe, controlled shutdown of the circuits prior to a normalshutdown, de-energize process (e.g., de-energize a component(s)), and/orthe lock-out-tag-out process.

Additionally or alternatively, to facilitate adjusting the actualcontrols of the PLC or other logically controlled devices, the user canpresent voice commands (e.g., input to or captured by the audio sensorcomponent 204), and/or control or textual commands (e.g., input to thekey/control component 308) to the IMC 108. The IMC 108 can analyze thevoice commands and/or control or textual commands, and can determinecontrol information to facilitate adjusting the actual controls,switches, etc., of the PLC or other logically controlled devices. TheIMC 108 can communicate such control information (e.g., controlinstructions) to the PLC or other logically controlled devices via theconnection to facilitate adjusting the actual controls, switches, etc.,of the PLC or other logically controlled devices and, accordingly,shutting down the circuits.

Another issue that can arise is that parts can wear out after they areused or stressed in certain ways. While this data may be availableinside a controller or a database, it may be difficult to obtain by auser and may in fact not even be considered except when a given partfails. The ARD 102 can overcome these and other problems that may arisewith regard to obtaining desired data regarding the wearing out ofparts.

The IMC 108 of the ARD 102 can detect or determine the respective itemsof equipment in view or in the area of the user, based at least in parton sensor data obtained by the sensor component 104 and/or other datafrom the data store 114 or another data source(s), as more fullydescribed herein. The IMC 108 can correlate the respective items ofequipment in view of the lens component of the ARD 102 or in the area ofthe user with data from databases, machine historical data, or infraredthermal readings, etc., and can compare that data to lifetime ratings ofthe components (e.g., parts) of the equipment. For example, forrespective components of the equipment, the IMC 108 can analyze currentsensor data (e.g., infrared thermal measurements or images, voltagelevel measurements, current level measurements, . . . ) relating torespective components and obtained from one or more sensors, historicaloperational data and/or other pertinent data (e.g., respectivespecifications, respective lifetime ratings, respective maximum orminimum parameters, . . . ) regarding the respective components (e.g.,obtained from the data store 114 or another data source(s)). Thehistorical operational data can comprise or relate to, for example,on-off cycles, voltages, thermals, current range, instances of exceedinga rating or threshold, the length of time the component has been inactive service, and/or other pertinent historical operational data withregard to the respective components.

Based at least in part on the results of such analysis, the IMC 108 candetermine the respective wear or stress levels of the respectivecomponents, and can determine respective “risk” levels (e.g., riskvalues) of the respective components, wherein a risk level of acomponent can comprise, or be determined based on, the risk of failureof the component, the risk that the failure of the component can resultin a failure of or damage to another component, the cost associated withthe failure of the component, a hazard associated with the failure ofthe component, the loss of production resulting from the failure of thecomponent, and/or another factor(s), in accordance with the definedinformation management criteria. The IMC 108 can generate customizedinformation (e.g., augmented and/or virtual reality information) thatcan comprise a visual presentation that can include a virtual display ofthe respective risk levels of the respective components in the field ofview of the user as the user wears the ARD 102.

The IMC 108 can facilitate presenting (e.g., overlaying), via thedisplay screen of the interface component 106, the visual presentationcomprising the virtual display of the respective risk levels of therespective components in the user's field of view to enable the user toascertain the respective risk levels of those respective components. Therespective risk levels can be positioned in the virtual display on thedisplay screen associated with the lens component of the ARD 102 tocorrespond to, and in proximity to, the location of the actual componentin the lens component of the ARD 102 as the user looks at the actualcomponent of the equipment through the lens component. This virtualdisplay can facilitate enabling the user to determine, with regard toeach of those components, whether a particular component is to bereplaced, repaired, or have maintenance performed on the particularcomponent, or whether another action is to be taken with regard to theparticular component.

Referring briefly to FIG. 6 (along with FIG. 1), FIG. 6 presents adiagram of an example visual presentation 600 that comprises a virtualdisplay that includes respective risk levels of respective components inthe field of view of the user through the ARD 102, in accordance withvarious aspects and embodiments of the disclosed subject matter. The IMC108 can generate the visual presentation 600 comprising the virtualdisplay, which can include respective risk levels of respectivecomponents in the field of view of the user, based at least in part onthe sensor data and/or other data, as more fully described herein.

In this example visual presentation 600, the virtual display cancomprise a first risk level 602 associated with a first component 604, asecond risk level 606 associated with a second component 608, and athird risk level 610 associated with a third component 612. The firstrisk level 602, the second risk level 606, and the third risk level 610can have respective risk level values, as determined by the IMC 108. Itis to be appreciated and understood that, while the visual presentation600 depicts three risk levels respectively associated with threecomponents, such example merely included three risk levels and threecomponents for reasons of brevity and clarity, and the visualpresentation 600 can include virtually any desired number of risk levelsassociated with respective components, based at least in part on thenumber of components in the field of view of the user of the ARD 102,wherein the number of risk levels can be three, more than three, or lessthan three.

The IMC 108 can position or locate the first risk level 602 in thevisual presentation 600, presented via the display screen of theinterface component 106, to the user to have the first risk level 602appear in proximity to the actual first component 604 as the user viewsthe actual first component 604 through the lens component of the ARD102. Similarly, the IMC 108 can position or locate the second risk level606 in the visual presentation 600, presented via the display screen ofthe interface component 106, to the user to have the second risk level606 appear in proximity to the actual second component 608 as the userviews the actual second component 608 through the lens component of theARD 102; and the IMC 108 can position or locate the third risk level 610in the visual presentation 600, presented via the display screen of theinterface component 106, to the user to have the third risk level 610appear in proximity to the actual third component 612 as the user viewsthe actual third component 612 through the lens component of the ARD102.

To facilitate determining the wear or stress levels associated withcomponents, determining respective risk levels of respective components,and/or determining whether components are to be replaced, be repaired,or have maintenance performed on such components, the IMC 108 canidentify and/or obtain statistics regarding how often respectivecomponents are or were activated, on/off cycles of a relay over itslifetime vs. the rating of a relay with regard to on/off cycles,voltage, thermal, and/or current range history for a given node, and/orother statistics (e.g., other current or historical operational data)associated with respective components. The IMC 108 can obtain suchstatistics from the data store 114 and/or another data source(s) (e.g.,a database(s)).

The IMC 108 can facilitate presenting, via the interface component 106,such statistics to the user for the user to review or analyze to enablethe user to facilitate determining the wear or stress levels associatedwith components and/or determining whether components are to bereplaced, be repaired, or have maintenance performed on such components.Additionally or alternatively, the IMC 108 can analyze such statistics,and, based at least in part on the analysis results, can determine therespective wear or stress levels associated with respective componentsand/or determine whether respective components are to be replaced, berepaired, or have maintenance performed on such components.

Yet another issue that can arise in connection with performingmaintenance work on equipment can relate to determining which cabinet ina plant or which part in a system (e.g., industrial automation system)contains a serviceable part. Determining which cabinet in a plant and/orwhich part in a system contains a serviceable part can be prone tomistakes, particularly when there are many similar cabinets in the plantor many similar parts in the system.

To facilitate determining which cabinet in a plant or which part (e.g.,component or other portion) in a system contains a serviceable part, thesensor component 104 of the ARD 102 can employ one or more sensors thatcan sense the location and/or orientation of the user and/or thelocation and/or orientation of equipment (e.g., cabinet, device, unit, .. . ), as more fully described herein. The sensors of the sensorcomponent 104 can employ the location sensor (e.g., 206) (e.g., GPS)and/or the orientation sensor (e.g., 208) to facilitate sensing thelocation and/or orientation of the user and/or the location and/ororientation of the equipment. The image capture component 202, scannercomponent 210, recognition component 212, and/or another sensor(s) 216of the sensor component 104 can be employed to capture images of parts(e.g., capture identification information (e.g., part code) on a part),scan respective part codes (e.g., scan a bar code) associated withrespective parts, perform object recognition on parts to facilitateidentifying or recognizing respective parts or respective part codesassociated therewith, and/or other sensing to generate sensor data thatcan facilitate identifying parts in a plant or system.

The IMC 108 can analyze the sensor data and/or other data, wherein theother data can be obtained from the data store 114 or another datasource(s) (e.g., database(s)). Based at least in part on the results ofsuch data analysis, the IMC 108 can identify respective parts, identifyrespective locations and/or orientations of respective parts, determinerespective parts that are serviceable, and determine (e.g., make apositive determination regarding) whether a particular cabinet orparticular portion of a system contains a serviceable part.

The IMC 108 can facilitate presenting, via the interface component 106,customized information (e.g., augmented and/or virtual realityinformation) to the user to notify or indicate to the user regardingwhich parts are serviceable parts, which cabinet or portion of thesystem contains a serviceable part, the location and/or orientation ofthe user, the location and/or orientation of the cabinet or portion ofthe system that contains the serviceable part, and/or other pertinentinformation. The customized information can be presented as visualinformation (e.g., image of the serviceable part, part code of the part,travel route to the part, . . . ) via display screen of the displaycomponent 302, as audio information (e.g., part code of the part, travelroute to the part, . . . ) via a speaker of the audio component 304,and/or as haptic feedback via the haptic component 306 (e.g., ARD 102can emit haptic feedback to the user when the user is in proximity tothe serviceable part and/or associated cabinet or system portion).

To guide the user to the serviceable part, the customized informationalso can comprise a travel route from the user's current location to thelocation of the cabinet or portion of the system that contains aserviceable part. The customized information further can includeidentification information (e.g., image of the part, part code of thepart, visual indicator that highlights the part on the display screen, .. . ) that can identify the serviceable part to the user, so that theuser can recognize the part when the user sees it (e.g., inside or onthe cabinet, or inside or on a component or device associated with thesystem portion) and can distinguish it from other parts. The customizedalso can comprise cabinet or system portion identification information(e.g., image of the cabinet or system portion, identifier or code thatidentifies the cabinet or system portion, visual indicator thathighlights the cabinet or system portion on the display screen, . . . )that can identify the cabinet or system portion that contains theserviceable part to the user, so that the user can recognize the cabinetor system portion when the user sees it.

As disclosed herein, other aspects, embodiments, and techniques relatingto using the ARD 102 to assist in (e.g., to perform or facilitateperforming (e.g., enabling or enhancing performance of)maintenance-related activities or tasks can comprise the same or similaraspects, embodiments, and techniques relating to using the ARD 102 toassist in (e.g., to perform or facilitate performing (e.g., enabling orenhancing performance of) assembly-related activities or tasks, as morefully described herein.

The ARD 102 also can similarly be used for enhancing design ofindustrial machines and systems as well as HMI design activities (e.g.,industrial and HMI design assistance). Various problems may arise duringindustrial machine and HMI design. For example, keeping track ofmultiple layers of data regarding parts (e.g., electrical nodes,connection points, input/output (I/O) points, ducts, gears, pumps,buttons, . . . ) during machine design and HMI design can beoverwhelming and/or burdensome to users, and can be prone to error byusers. The looking up of data also can be time consuming and burdensome.As another example, determining reliability of a design by keepingcomponents within manufacturer's guidelines can be time consuming andunintuitive. As yet another example, determining functional safety canbe done, in part, by considering what-if scenarios at each node andpart. This can be undesirably (e.g., exceedingly) time consuming andunintuitive. As still another example, visualizing thermal and crosstalkissues at a system level can be difficult. Further, in attempting to fitall pertinent information and control on a physical HMI screen, it canbe difficult to maintain viable and/or readable sizes of theinformation. Moreover, communicating system hazards, warnings, and tipsto end users often can be limited by having to use undesirably smalllabels and user manuals.

In accordance with various implementations, with the user (e.g.,operator) interacting with or wearing the ARD 102, the ARD 102 canassist the user in a number of ways, such as disclosed herein, toimprove speed of the user and of the design of industrial machines orsystems and/or design of HMIs, quality of design of industrial machinesor systems and/or design of HMIs, security of design of industrialmachines or systems and/or design of HMIs, and safety of the design ofindustrial machines or systems and/or design of HMIs. The ARD 102 canadvise or instruct the user through visual signals, audible signals,and/or haptic feedback to facilitate enhancing the design process (e.g.,enhancing, the speed, quality, security, and safety associated with thedesign process).

As stated, keeping track of multiple layers of data regarding parts(e.g., electrical nodes, wires, connection points, I/O points, ducts,gears, pumps, buttons, . . . ) during machine design and HMI designpotentially can be overwhelming or burdensome, and can be prone toerror. Further, it can be undesirably time consuming for a user to haveto look up data in connection with machine design or HMI design.

To overcome these and other issues, the IMC 108 of the ARD 102 candetermine, generate, and provide multiple layers of data about any point(e.g., device) of a system (e.g., industrial automation system) bypresenting the information regarding or relating to such device of thesystem. The IMC 108 can keep track of multiple layers of data regardingrespective parts (e.g., electrical nodes, wires, connection points, I/Opoints, ducts, gears, pumps, buttons, . . . ) during machine design andHMI design. The IMC 108 can determine and generate, and can facilitatepresenting to the user (e.g., designer), via the display screen of theARD 102, desired information (e.g., enhanced information) regardingdevices, wires, I/O points, etc., of a system, wherein the ARD 102 canoverlay the desired (e.g., enhanced) information on the actual/livescene being perceived by the user through the lens component of the ARD102.

The IMC 108 can receive or obtain data (e.g., identification data,configuration data, current and/or historical operational data, datarelating to conditions, specifications, user manuals, warning data orindicators, . . . ) regarding respective devices, nodes, components,wires, or points of the system from sensors of the sensor component 104and/or other data obtained from the data store 114 or other sources(e.g., external databases). To access and view data regarding aparticular device (or node, component, wire, or point of the system),the user can select the particular device (or node, component, wire, orpoint of the system), for example, by viewing the particular device viathe live view (e.g., via the lens component) of the ARD 102 and/or viasome other technique of selecting the device (e.g., making a handgesture that can be captured by the image capture component 202 toselect the device; issuing a voice command to the ARD 102, via the audiosensor component 204, to select the device; or selecting a control onthe ARD 102, via the key/control component 308). In response, the IMC108 can identify and facilitate presenting, via the interface component106 (e.g., via the display screen of the interface component 106), atleast a portion of the data (e.g., device identification data, summaryof the data, current data, most pertinent data, and/or warning data, . .. ) associated with the particular device. It is to be appreciated andunderstood that, while the example embodiments relate to accessing andviewing data associated with a particular device, the user can employthe ARD 102 to access and view desired data with regard to any device,node, component, wire, or point of the system.

As desired, the user can select which additional information (e.g.,regarding the particular device or the system) that the user wants toview (e.g., on the display screen of the ARD 102) using, for example,verbal instructions issued to the ARD 102, via the audio sensorcomponent 204, or using defined gestures (e.g., hand, eye, or facialgestures) that can be captured by the image capture component 202. Suchinformation associated with the particular device can include, forexample, tag names at each I/O point in the design, configurationinformation regarding components, parts, or points, of the system,voltages (e.g., in numerical format, oscilloscope format, heat-mapformat, . . . ), frequencies, Internet Protocol (IP) addresses, flowrate, datasheets, schematic diagrams, ladder diagrams, CAD drawings,range of setpoints, expected values, or possible (e.g., historical)values for a given node, and/or other items of data of the portion ofthe data, etc.

For instance, the user can make a first type of hand gesture to selectan item(s) of the data associated with the particular device forviewing. The image capture component 202 can capture that first type ofhand gesture. In response to the first type of hand gesture, the IMC 108can identify and/or obtain the item(s) of the data and can facilitatepresenting, via the display screen of the interface component 106(and/or via the audio component 304), the selected item(s) of the datato the user for viewing. As desired, the user can make a second type ofhand gesture to drill down and select more detailed data associated withthe particular device for viewing (e.g., drill down from one layer ofdata to another layer(s) of data). The image capture component 202 cancapture that second type of hand gesture. In response to the second typeof hand gesture, the IMC 108 can identify and/or obtain the moredetailed data and can facilitate presenting, via the display screen ofthe interface component 106 (and/or via the audio component 304), themore detailed data to the user for viewing.

Another issue that can arise is that determining reliability of amachine design or HMI design by maintaining components withinmanufacturer's guidelines can be time consuming and unintuitive. Thedisclosed subject matter can overcome these and other problems relatingto machine design and HMI design.

The IMC 108 of the ARD 102 can access design documents and manufacturerdatasheets associated with a system (e.g., devices, processes, assets, .. . , of the system). The IMC 108 also can receive sensor data from thesensors of the sensor component 104, wherein the sensor data can relateto conditions, parameters, device or process configurations,interrelationships between devices or nodes, and/or other features ofthe system. For instance, the sensor component 104, another device,and/or the user(s) can perform live measurements of conditions andparameters associated with devices, processes, and/or assets, etc., ofthe system (e.g., industrial automation system), wherein the datarelating to the live measurements can be received by the IMC 108.

The IMC 108 can analyze the data obtained from the design documents andmanufacturer datasheets, the sensor data, and/or live measurement dataassociated with the system. Based at least in part on the data (e.g.,raw data) and/or the results of analyzing the data, the IMC 108 cangenerate a visual presentation, comprising a color enhancement, and/orother visual cues (e.g., visual indicators) that can report and indicatewhich devices within the user's field of view through the lens componentof the ARD 102, or otherwise in proximity to the user, are determined tobe close to or are exceeding manufacturer ratings, other pertinentthreshold values (e.g., pertinent threshold ratings), and/or otherattributes of the devices of the system. The IMC 108 can facilitatepresenting (e.g., overlaying), via the display screen of the interfacecomponent 106, the visual presentation, comprising the color enhancementrelating to the devices, to the user, wherein the user can view theinformation in the visual presentation on the display screen as the userperceives the devices in the user's field of view through the lenscomponent of the ARD 102.

Referring to FIG. 7 (along with FIG. 1), FIG. 7 depicts a diagram ofvarious example visual presentations 700 comprising data (e.g.,pertinent data) and/or visual indicators relating to components (e.g.,devices, nodes, connections, . . . ) of a system that are in the fieldof view of (or otherwise in proximity to) the user through the ARD 102,in accordance with various aspects and embodiments of the disclosedsubject matter. The example visual presentations 700 can comprise, forexample visual presentation 702 that can comprise a color enhancementand/or other visual cues (e.g., visual indicators) relating to devicesof a system that are in the field of view of (or otherwise in proximityto) the user through the ARD 102, in accordance with various aspects andembodiments of the disclosed subject matter. The IMC 108 can generatethe visual presentation 702, based at least in part on the results ofanalyzing the sensor data and/or the other data, in accordance with thedefined information management data and/or preferences of a user, asmore fully described herein.

If, for example, the IMC 108 determines that a first device 704 is orhas been close to exceeding a manufacturing rating or another pertinentthreshold value associated with the first device 704, the IMC 108 canutilize apply a first color enhancement or visual cue 706 having a firstcolor (e.g., yellow) to indicate that the first device 704 is or hasbeen close to exceeding the manufacturing rating or other pertinentthreshold value associated with the first device 704. As the userperceives the first device 704 (e.g., the actual first device) in theuser's field of view through the lens component of the ARD 102, the IMC108 can facilitate presenting, via the display screen of the interfacecomponent 106, the first color enhancement or visual cue 706 having thefirst color over or in proximity to the first device 704 in the field ofview of the user.

As another example, if the IMC 108 determines that a second device 708(e.g., in the user's field of view) is or has exceeded a particularmanufacturing rating or other pertinent threshold value associated withthe second device 708, the IMC 108 can utilize apply a second colorenhancement or visual cue 710 having a second color (e.g., red) toindicate that the second device 708 is or has exceeded the particularmanufacturing rating or other pertinent threshold value associated withthe second device 708. As the user perceives the second device 708(e.g., the actual second device) in the user's field of view through thelens component of the ARD 102, the IMC 108 can facilitate presenting,via the display screen of the interface component 106, the second colorenhancement or visual cue 710 having the second color over or inproximity to the second device 708 in the field of view of the user.

As still another example, if the IMC 108 determines that a third device712 (e.g., in the user's field of view) has not been close to exceedingand has not exceeded a particular manufacturing rating or otherpertinent threshold value associated with (e.g., applicable to) thethird device 712, the IMC 108 can utilize apply a third colorenhancement or visual cue 714 having a third color (e.g., green) toindicate that the third device 712 has not been close to exceeding andhas not exceeded the particular manufacturing rating or other pertinentthreshold value associated with the third device 712. As the userperceives the third device 712 (e.g., the actual third device) in theuser's field of view through the lens component of the ARD 102, the IMC108 can facilitate presenting, via the display screen of the interfacecomponent 106, the third color enhancement or visual cue 714 having thethird color over or in proximity to the third device 712 in the field ofview of the user.

The ARD 102, by generating and presenting the visual presentation 702,can thereby enable the user to quickly visualize the respectiveattributes of respective devices of the system. This can lead to orresult in improving the reliability and lifetime of the devices of thesystem. It is to be appreciated and understood that, while the aboveexample visual presentation 702 utilizes three particular colors withrespect to three different types of conditions associated with threedevices, the disclosed subject matter is not so limited. For instance,the IMC 108 can determine and generate visual presentations comprisingcolor enhancements that can employ a higher level of granularity withregard to respective conditions or attributes of respective devices of asystem, including employing more and/or different colors, more and/ordifferent shades of colors, other types of visual indicators besides orin addition to colors, and/or can provide data in desired detail or canfacilitate enabling the user to drill down into the visual presentation702 (e.g., by selecting a device) to obtain additional data (e.g., otherdata layers comprising other data), in accordance with the definedinformation management data and/or preferences of a user.

With regard to derating and reliability, the IMC 108 of the ARD 102 alsocan compare these factors (e.g., manufacturing ratings, other pertinentthreshold values, and/or other attributes) associated with devices ofthe system to respective corporate or industry derating guidelinesrelating to the devices. For instance, the IMC 108 can compare thestatistics (e.g., voltage, frequency, current, load, flow, . . . )associated with the devices (or nodes, components, points, . . . ) ofthe system to manufacturing datasheets and engineering (e.g.,good-engineering) or corporate derating guidelines. The IMC 108 candetermine differences between the factors associated with the respectivedevices and the respective corporate or industry (e.g., engineering)derating guidelines relating to the respective devices, based at leastin part on the results of the comparison, wherein the comparison resultscan relate to derating and reliability associated with the system. TheIMC 108 can facilitate presenting, to the user, via the display screenof the ARD 102, the comparison results relating to derating andreliability associated with the system.

The IMC 108 also can determine respective color enhancements or visualcues for respective ratings, threshold values, and/or other attributesassociated with respective devices of the system based at least in partthe respective corporate or industry derating guidelines relating to therespective devices. The IMC 108 can facilitate presenting, via thedisplay screen of the interface component 106 the respective colorenhancements or visual cues associated with respective devices withrespect to the respective corporate or industry derating guidelines, inproximity to (e.g., alongside of) the other color enhancements or visualcues (e.g., 706, 710, 714) associated with the devices, or in a separatevisual presentation, wherein the user can select (e.g., using gestures,verbal commands, textual or control commands) between presentation ofthe respective visual presentations, or respective portions (e.g., on adevice basis) thereof, on the display screen in the field of view of theuser.

Another problem that can arise can relate to what-if scenarios inconnection with a machine design or HMI design. For instance,determining functional safety can be done, in part, by consideringwhat-if scenarios at each node and part. However, this can beundesirably time consuming and unintuitive. The disclosed subject mattercan overcome these and other issues relating to machine design or HMIdesign.

In some implementations, the user, using the ARD 102, can select one ormore nodes, or all nodes (and parts), on which the IMC 108 can performwhat-if scenarios (e.g., safety what-if scenarios) to check thefunctional safety of the design of the system. The user can use the ARD102 to test various desired attributes or conditions associated with thedesign of the system, wherein the desired attributes or conditions cancomprise, for example, overvoltage, overcurrent, fail open, fail closed,overheat, stuck bit, etc., associated with a node or part of the system.The IMC 108 can maintain data to track what was tested with regard tothe nodes or part of the system by the ARD 102, and what portions (e.g.,other nodes or parts) of the system remain to tested on a checklist(e.g., a virtual checklist matrix).

The IMC 108 can analyze the data obtained from the design documents andmanufacturer datasheets, the sensor data, and/or live measurement dataassociated with the system. Based at least in part on the data (e.g.,raw data) and/or the results of analyzing the data, the IMC 108 cangenerate what-if scenarios (e.g., safety what-if scenarios) performed onselected nodes of the system. The IMC 108 can facilitate presenting, tothe user, via the display screen of the interface component 106,scenario result information regarding the results of the what-ifscenarios performed on the selected nodes of the system, in accordancewith the defined information management criteria and/or userpreferences.

The IMC 108 also can determine possible or potential what-if scenariosassociated with respective nodes of the system. The IMC 108 can maintainand track a list of the possible or potential what-if scenariosassociated with certain respective nodes, and can facilitate presenting,to the user, via the display screen of the interface component 106, thelist (e.g., hovering or virtual checklist) of the possible or potentialwhat-if scenarios associated with respective nodes. The possible orpotential what-if scenarios associated with respective nodes can be orrelate to, for example, an overvoltage condition, an overcurrentcondition, a fail open condition, a fail closed condition, an overheatcondition, a stuck bit, and/or another condition(s) associated with therespective nodes.

The IMC 108 of the ARD 102 also can compare such what-if scenariofactors (e.g., scenario results) or conditions to applicable corporateor industry derating guidelines associated with respective devices ornodes. The IMC 108 can determine differences between the what-ifscenario factors or conditions associated with the respective nodes ordevices and the respective corporate or industry derating guidelinesrelating to the respective nodes or devices, based at least in part onthe results of the comparison. Based at least in part on the comparisonresults, the IMC 108 can generate and facilitate presenting, via thedisplay screen of the interface component 106, a visual presentationthat can comprise the results of the comparison of the what-if scenariofactors or conditions associated with the respective nodes or deviceswith the respective corporate or industry derating guidelines associatedwith the respective nodes or devices.

By employing the disclosed techniques and features, the ARD 102 canthereby assist the user in intuitively visualizing the system (e.g., thedevice and nodes of the system) and data relating to the system, whilethe ARD 102 can perform the what-if scenarios and can maintain data thatcan track the completeness of the analysis of data (e.g., on a virtualchecklist), including the results of what-if scenarios, and/or whatportions of the system remain to be tested or evaluated (e.g., byperforming a what-if scenario), in the data store 114 or on anotherdevice (e.g., a computer, a server, an external data store, . . . )associated with (e.g., communicatively connected to) the ARD 102 (e.g.,via a wireless or wireline communication connection).

In connection with the what-if scenarios, the user of the ARD 102 canemploy the ARD 102 to make various adjustments to a system, performvarious tests, or perform other desired tasks. For instance, the usercan use the ARD 102 to manipulate a virtual component (e.g., anas-yet-unplaced virtual component) associated with the system tofacilitate determining a desired (e.g., enhanced, optimal, suitable, oracceptable) placement of the virtual component in relation to otherportions (e.g., node, wire, connection, I/O point, . . . ) of the systemwith respect to wire length, thermal considerations, crosstalk, or otherdesired attributes or characteristics, in accordance with the definedinformation management criteria and defined design criteria. The virtualcomponent can correspond to (e.g., model) an actual component of thesystem. The user can employ hand gestures, voice commands, or text orcontrol commands, for example, which can be received or detected by theARD 102, as disclosed herein, to facilitate manipulating the placementof the virtual component.

In response to the manipulation of the virtual component to adjust theplacement of the virtual component by the user, the IMC 108 can performtests (e.g., what-if scenarios) on desired (e.g., selected) or relevantnodes or points of the system to facilitate determining whether the new(e.g., adjusted) placement is desirable (e.g., enhanced, optimal,suitable, or acceptable). If, based at least in part on the testresults, the new placement of the virtual component is desirable, asdetermined by the user or the IMC 108, the user can use the ARD 102 toselect the new placement of the virtual component for the design. If,based at least in part on the test results, the new placement of thevirtual component is not desirable, as determined by the user or the IMC108, the user can use the ARD 102 to manipulate the virtual component toadjust the placement of the virtual component, wherein desired testingcan be performed by the IMC 108 based on the new placement.

As an example, the user can use the ARD 102 to manipulate placement of avirtual fan, corresponding (e.g., modeling) an actual fan of the system,and virtual fan speed (e.g., corresponding to actual fan speed), toachieve a desirable (e.g., enhanced, optimal, suitable, or acceptable)heat flow with regard to a system. The ARD 102 can be employed toperform testing and/or further adjustments to placement of the virtualfan and/or fan speed, based at least in part on test results, tofacilitate determining a desired placement of the virtual fan in thesystem and a desired fan speed, in accordance with the definedinformation management criteria and defined design criteria.

As another example, the user can utilize the ARD 102 to test one or morecomponents of the system by forcing or toggling states using one or morevirtual controls generated by the IMC 108 or another device, which canprovide the one or more virtual controls to the ARD 102. For instance,the IMC 108 can generate one or more virtual controls for a system,based at least in part on the results of analyzing data, such as sensordata received from the sensor component 104 and/or other data relatingto the system or controls obtained from the data store 114 or anotherdata source(s) by the IMC 108. The user and/or the IMC 108 canmanipulate the one or more virtual controls to force or toggle states ofone or more components of the system to obtain test results relating tothe response of the one or more components to such controlmanipulations. The test results can be used by the user or IMC 108 tofacilitate determining whether modifications are to be made to the oneor more components with respect to the design.

Yet another issue that can arise in connection with machine design orHMI design can relate to thermal and crosstalk issues associated with adesign. For instance, visualizing thermal and crosstalk issues at asystem level can be difficult for users. The disclosed subject matter,employing the ARD 102, can mitigate or overcome issues relating tothermal and crosstalk issues and/or other problems associated withmachine design or HMI design.

As disclosed herein, the ARD 102 can be utilized to generate and present(e.g., in real time or substantially real time) visual presentationsthat can include color or other visual enhancements (e.g., a color-codedheat map) that can visualize temperatures (e.g., IR temperatures ofvarious portions of the system obtained using the temperature sensorand/or the image capture component 202 and translated to visible, visualdepictions or indicators) of a system, as perceived through the sensorcomponent 104 of the ARD 102. By being able to correlate the system inreal time, or at least substantially real time, and multi-dimensional(e.g., 3-D) viewing (e.g., via the heat map), the ARD 102 can make iteasier for the user to understand and correct issues (e.g., temperatureissues, unacceptable component operation, and/or abnormal parameters, .. . ) relating to components (e.g., devices, connections, wires, . . . )of the system.

The ARD 102 also can extend these aspects, as the ARD 102 also canvisualize high frequencies, high current, and/or other physical (butinvisible) attributes associated with components of the system that cancause crosstalk or mutual interference between components of the system.For instance, the IMC 108 can receive sensor data from sensors of thesensor component 104 and/or other data from the data store 114 oranother data source(s). The IMC 108 can analyze the sensor data and/orthe other data. Based at least in part on the results of the analysis,the IMC 108 can determine respective temperatures associated withrespective components (e.g., devices, connections, wires, . . . ) of thesystem, including components (e.g., internal components of a cabinet)that are not readily visible to the user, and can translate therespective temperatures to respective visual representations orindicators, in accordance with the defined information managementcriteria and/or user preferences.

Further, based at least in part on the results of the analysis, the IMC108 can determine respective frequencies, current, and/or other physicalattributes (including attributes that are not visible) associated withthe respective components (e.g., devices, connections, wires, . . . ) ofthe system, including components (e.g., internal components of acabinet) that are not readily visible to the user. The IMC 108 cantranslate the respective frequencies, current, and/or other physicalattributes associated with the respective components to respectivevisual representations or indicators, in accordance with the definedinformation management criteria and/or user preferences. The IMC 108 cangenerate a visual presentation that can comprise a heat map and/or acrosstalk and interference map, based at least in part on the respectivevisual representations or indicators relating to the temperatures and/orthe respective visual representations or indicators relating to therespective frequencies, current, and/or other physical attributes. Forinstance, the heat map can depict respective visual representations orindicators associated with the respective components of the system, andthe crosstalk and interference map can depict respective visualrepresentations or indicators associated with the respectivefrequencies, current, and/or other physical attributes associated withthe respective components.

The IMC 108 can position the various respective visual representationsor indicators in the visual presentation such that, when displayed onthe display screen, the various respective visual representations orindicators can be overlaid and displayed in proximity to the actualcomponents of the system in the field of view of the user as the userviews the actual components through the lens component of the ARD 102.In some implementations, the IMC 108 can facilitate presenting the heatmap and the crosstalk and interference map on the display screensimultaneously (e.g., when in accordance with the defined informationmanagement criteria and/or user preferences). In other implementations,the IMC 108 can facilitate presenting the heat map and the crosstalk andinterference map on the display screen separately at differentinstances, as the respective maps are selected by the user, for example,via hand gestures, voice commands, or control or key commands (e.g.,when in accordance with the defined information management criteriaand/or user preferences). As the user moves and the user's field of viewis changed accordingly, and/or if conditions associated with thecomponents of the system change, the IMC 108 can update (e.g., modify inreal time or substantially real time) the heat map and/or the crosstalkand interference map to account for and reflect the changes in theuser's field or view and/or the change in conditions of the components.

The visual representations or indicators of the heat map can be similarto the visual representations or indicators employed in the heat map 504of FIG. 5. The visual representations or indicators relating to therespective frequencies, current, and/or other physical attributesassociated with the respective components that are presented in thecrosstalk and interference map can be relatively similar in some ways tothe visual representations or indicators employed for the heat map, butcan be different overall to facilitate differentiating between thevisual representations or indicators used in the heat map from thevisual representations or indicators used in the crosstalk andinterference map. For example, the visual representations or indicatorsused in the crosstalk and interference map can use different colors (orno color), different markings (e.g., a region with a particular (e.g.,unique) line pattern running across the region, wherein the region canhave a particular color or no color), different sizes (e.g., a visualrepresentation relating to temperature can cover a region having a firstsize associated with that temperature, and a visual representationrelating to frequencies, current, and/or other physical attributes canhave a different (e.g., smaller) size that is sufficient to point outthe component associated with such visual representation, withoutobstructing the visual representation relating to temperature), and/orother different characteristics.

Referring again briefly to FIG. 7 (along with FIG. 1), FIG. 7 furthercomprises example visual presentation 720 that can comprise a heat mapand a crosstalk and interference map associated with the system, inaccordance with various aspects and embodiments of the disclosed subjectmatter. As depicted in the visual presentation 720 (e.g., in the fieldof view of the user), the visual presentation 720 can comprise a firstcomponent (comp.) 722, second component 724, and third component 726,which can be arranged in relation to each other and/or connected to eachother, in accordance with the design. With respect to the heat mapportion of the visual presentation 720, the IMC 108 can facilitatepresenting a first visual representation or indicator 728 having a firstcolor (e.g., red) to represent a high temperature region associated withthe first component 722, a second visual representation or indicator 730having a second color (e.g., yellow) to represent a moderate temperatureregion associated with the second component 724, and a third visualrepresentation or indicator 732 having a third color (e.g., blue) torepresent a relatively cooler temperature region associated with thethird component 726. The respective temperatures of the heat map can bedetermined by the IMC 108, based at least in part on data analysisresults, as disclosed herein. Other temperature regions of the field ofview have been omitted for reasons of brevity and clarity.

With respect to the crosstalk and interference map portion of the visualpresentation 720, the IMC 108 can facilitate presenting a fourth visualrepresentation or indicator 734 having a first pattern to represent aregion having a high current associated with the first component 722,and a fifth visual representation or indicator 736 having a secondpattern to represent another region having a high current associatedwith the second component 724. The IMC 108 can determine the highfrequency region and the high current region, based at least in part ondata analysis results, as disclosed herein. As depicted in the visualpresentation 720, the visual representations or indicators (e.g., 728,730, 732) associated with the heat map portion can be of a differenttype than the visual representations or indicators (e.g., 734, 736)associated with the crosstalk and interference map portion of the visualpresentation 720. This can facilitate enabling the user to differentiatebetween the different types of visual representations or indicators andcorresponding data.

In some implementations, the IMC 108 can employ different visualrepresentations or indicators for different attributes (e.g.,frequencies, current, other physical attributes). In certainimplementations, the IMC 108 can determine and generate the heat mapand/or the crosstalk and interference map such that respective visualrepresentations or indicators presented in the respective maps can belimited to present only those respective visual representations orindicators that are a problem or potentially a problem (e.g., satisfy(e.g., meet or exceed) a defined temperature threshold level, satisfy adefined threshold relating to crosstalk or interference). This canfacilitate the presentation of what is typically more pertinentinformation, without overwhelming the user with too much information(e.g., information regarding components that are at room temperature orother temperature that is not noteworthy, or information regardingcomponents for which there is little or no risk of crosstalk orinterference).

In still other implementations, the IMC 108 can generate and facilitatepresenting different crosstalk and interference maps, based at least inpart on different attributes associated with the respective componentsof the system, in accordance with the defined information managementcriteria and/or user preferences of the user. For example, the IMC 108can generate and facilitate presenting a first crosstalk andinterference map, based at least in part on a first attribute (e.g.,frequencies), and a second crosstalk and interference map, based atleast in part on a second attribute (e.g., current levels).

With further regard to the crosstalk and interference map, the ARD 102,by presenting the visual presentation, comprising the crosstalk andinterference map that can visualize the respective frequencies, current,and/or other physical attributes associated with the respectivecomponents of the system, can enable the user viewing such map toidentify and make decisions regarding high frequencies, high current,and/or other physical (but potentially invisible) attributes that cancause or potentially cause unacceptable cross talk or mutualinterference issues.

Another potential issue that can arise in connection with machine designor HMI design can relate to putting a significant amount of data on adisplay screen. For example, when fitting or attempting to fit allpertinent information and control on a physical HMI screen, it may bedifficult to maintain viable and/or readable sizes that can beascertained or readily ascertained by a user.

The user can use the ARD 102 to facilitate identifying components, I/Opoints, etc., associated with the system by using the ARD 102 to focusand select the components, I/O points, etc. This can make it isrelatively easier for the user of the ARD 102 to construct a logicroutine and also HMI screens that can use them and/or other features orcomponents of a machine or HMI design.

For instance, the IMC 108 can analyze sensor data relating to sensedconditions associated with the system from the sensor component 104and/or other data relating to the system from the data store 114 oranother data source(s). The IMC 108 can determine customizedinformation, comprising augmented and/or reality information, that caninclude a virtual representation of the components, I/O points, etc., ofthe system (e.g., virtual components, virtual I/O points, etc., of thesystem), based at least in part on the results of the analysis of thesensor data and/or other data. The IMC 108 also can determine othercustomized information, comprising other augmented and/or realityinformation, that can include a virtual representation of HMIs,including virtual HMI components (e.g., measurements, controls, . . . )associated with the system, based at least in part on the results of theanalysis of the sensor data and/or other data.

The IMC 108 can facilitate presenting, to the user, via the displayscreen of the interface component 106, the customized information thatcan include the virtual representation of the components, I/O points,etc., of the system. The user can use the ARD 102 to focus, magnify,and/or select desired virtual components, virtual I/O points, and/orother virtual objects of the virtual representation of the system. Forexample, based at least in part on a hand gesture of the user (e.g.,captured by the outward-facing camera of the image capture component202), focus or movement of the eye of the user (e.g., captured by theinward (user)-facing camera of the image capture component 202), a voicecommand (e.g., received via the audio sensor component 204), or textualor control command (e.g., received via the key/control component 308),the IMC 108 can determine that the user desires to focus or magnify aportion of the virtual representation of the system, comprising certainvirtual components, virtual I/O points, or other virtual objects. Inresponse, the IMC 108 can emphasize and/or magnify the portion of thevirtual representation to bring it into enhanced (e.g., sharper) focus,and/or can de-emphasize and/or reduce in size another portion of thevirtual representation. In some implementations, the de-emphasizing ofthe other portion of the virtual representation can comprise a blurringof the other portion of the virtual representation.

Referring to FIG. 8 (along with FIG. 1), FIG. 8 illustrates a diagram ofvarious example visual presentations 800 that can facilitate enabling auser to identify, focus on, and/or select desired items of data (e.g.,components, controls, . . . ) associated with a system, in accordancewith various aspects and embodiments of the disclosed subject matter.The example visual presentations 800 can comprise, for example visualpresentation 802 that can comprise a virtual representation ofcomponents, I/O points, etc., of the system (e.g., virtual components,virtual I/O points, etc., of the system), based at least in part on theresults of the analysis results. The visual presentation 802 cancomprise various example components and I/O points, wherein thecomponents can comprise, for example, components 804, 806, 808, 810,812, and 814 that can be arranged in relation to each other and/orassociated with each other via connections, in accordance with thedesign.

The user of the ARD 102 may desire to focus on components 806 and 812for some reason. In accordance with various aspects, in response to acommand (e.g., gesture-related command, eye-related command, verbalcommand, or key or control command) received by the IMC 108 thatindicates that the user wants to focus on components 806 and 812, theIMC 108 can modify the visual presentation 802 to generate visualpresentation 820 to enhance the focus or emphasis on the components 806and 812. For instance, as depicted in the visual presentation 820, thecomponents 806 and 812, and associated connections, within the window822 have been enlarged (e.g., magnified) relative to the othercomponents (e.g., 804, 808, 810, and 812). This can enhance the focus onthese components 806 and 812 for the user, enable significantly moreinformation to be displayed in the window 822 (e.g., area of focus),and/or can enable the user to see more details with regard to thesecomponents 806 and 812, and their associated connections.

In some implementations, the IMC 108 also can generate a virtualrepresentation of an HMI, including virtual HMI components (e.g.,measurements, controls, . . . ) associated with the system, based atleast in part on the results of the analysis of the sensor data and/orother data. With further reference to FIG. 8, visual presentation 840can comprise a virtual HMI that can comprise various virtual components,including various virtual controls and various items of data (e.g.,measurement data). The IMC 108 can generate the virtual HMI to have itcorrespond to an actual (e.g., physical) HMI of the system, wherein theactual HMI can comprise actual components (e.g., actual controls,measurement data) that can correspond to the virtual components of thevirtual HMI, and wherein the actual HMI can be associated with (e.g.,placed or installed on) a cabinet of the system. The IMC 108 also canconnect or facilitate connecting the virtual HMI to the actual HMI tofacilitate enabling the user to use the ARD 102 to manipulate thevirtual controls of the virtual HMI to correspondingly modify thecorresponding actual controls of the actual HMI.

As depicted in the visual presentation 840, the virtual HMI can comprisevirtual controls 842, 844, 846, and 848, and items of data 850, 852,854, and 856, which can be arranged in the virtual HMI to correspond tothe arrangement of the actual controls and items of data in the actualHMI. The user of the ARD 102 may desire to focus on virtual controls 846and 848 (which are smaller in size on the virtual HMI than the othervirtual controls 842 and 844) and/or item of data 856, for some desiredreason of the user. In accordance with various aspects of the disclosedsubject matter, in response to a command (e.g., gesture-related command,eye-related command, verbal command, or key or control command) receivedby the IMC 108 that indicates that the user wants to focus on virtualcontrols 846 and 848 and/or the item of data 856, the IMC 108 can modifythe visual presentation 840 to generate visual presentation 860 toenhance the focus or emphasis on the virtual controls 846 and 848 and/orthe item of data 856. For instance, as depicted in the visualpresentation 860, the virtual controls 846 and 848 and the item of data856, within the window 862 have been enlarged (e.g., magnified) relativeto the other virtual controls (e.g., 842, 844) and other items of data(e.g., 850, 852). Further, due to the size of the window 862, the itemof data 854 has been obscured and the virtual control has been partiallyobscured. The enhancing of the virtual controls 846 and 848 and the itemof data 856 can enhance the focus on the virtual controls 846 and 848and the item of data 856 for the user, enable more information to bedisplayed (and/or be more readable) in the window 862 (e.g., area offocus) with regard to the virtual controls 846 and 848 and the item ofdata 856, and/or can enable the user to see more details with regard tothe virtual controls 846 and 848 and the item of data 856.

The ARD 102, having the ability to place virtual HMI components (e.g.,measurements, controls, . . . ) anywhere within the user's field of viewas the user perceives the environment via the lens component of the ARD102, can enable significantly more information to be presented (e.g.,displayed) to the user, via the display screen of the interfacecomponent 106 of the ARD 102, up to and/or exceeding the entire surfaceof a cabinet (e.g., the electrical cabinet) where a physical HMI wouldotherwise have been placed.

Still another potential problem that can be associated with machinedesign or HMI design can involve the presentation of hazards, warning,tips, or other information to users. For instance, communicating systemhazards, warnings, and/or tips to end users often can be limited byhaving to use undesirably small labels and/or user manuals, which canresult in readability issues for end users.

To facilitate addressing and overcoming these and/or other issues, theARD 102 can be configured to encode warnings, hazard information, and/ortips (e.g., assistance, guidance) into the ARD 102. The IMC 108 canfacilitate presenting the warnings, hazard information, and/or tips tothe user of the ARD 102 merely in response to the sensor component 104(e.g., inward-facing camera of the image capture component 202) sensingthat the user is viewing the system or components, or portion thereof,through the lens component of the ARD 102 and/or in response to a handgesture command, verbal command, and/or key or control command receivedby the ARD 102 from the user.

The warnings, hazard information, tips, and/or other desired information(e.g., pertinent or valuable information) can be encoded into the ARD102 in text and/or as a visual representation, for example, in a way(e.g., virtual labels) that can be similar to actual labels (e.g.,actual warning labels, actual hazard labels, . . . ). The IMC 108 canfacilitate presenting (e.g., displaying) such virtual labels on thedisplay screen of the interface component 106 to correspond to thelocations of the devices in the field of view of the ARD user such thatthe virtual labels can appear to be projected on to the devices of thesystem as if the virtual labels are actual labels.

Additionally or alternatively, the ARD 102 (e.g., the IMC 108 of the ARD102) can encode the warnings, hazard information, tips, and/or otherdesired information in other desired ways. For example, the IMC 108 cangenerate or draw a ring or cast a color over a zone around a hazard(e.g., hazard keepout zones, such as arc-flash zones). When the user isin, or in proximity to, the zone, the sensor component 104 and/or IMC108 can detect the user's location relative to the location of the zone.In response, the IMC 108 can facilitate presenting, via the displayscreen of the interface component 106, the ring or colored region overthe zone comprising the hazard to facilitate notifying the user of thehazard.

Additionally or alternatively, the user can use the ARD 102 to tag anddefine hazards, hazard keepout zones, and/or safety critical regionsassociated with a system. In response, the IMC 108 can perform thetagging and defining of the hazards, hazard keepout zones, and/or safetycritical regions associated with the system. If the sensor component 104detects a tag associated with, for example, a hazard associated with thesystem, the IMC 108 can determine that the user is in proximity to thehazard, based at least in part on the sensor data relating to the tagfrom the sensor component 104. The IMC 108 can identify the hazard andits location in relation to the user based at least in part on thesensor data relating to the tag, sensor data relating to the location ofthe user, and/or other data. The IMC 108 can generate and facilitatepresenting, via the interface component 106, a warning indicator and/orwarning information to the user to warn the user of the hazard and itslocation, and/or to provide relevant information regarding the type ofhazard to the user. The warning indicator and/or warning information canbe presented, via the display screen of the display component 302, tothe user (and/or via audio indicator and/or information presented viathe audio component 304, and/or haptic feedback via the haptic component306).

Additionally or alternatively, the ARD 102 (e.g., the IMC 108 of the ARD102) can encode and/or present the warnings, hazard information, tips,and/or other desired information in still other desired ways that can bemore than or other than visual. For example, the IMC 108 can generateaudible indicators and/or haptic indicators that can be associated withthe zone and/or hazard. In response to detecting that the user is in, orin proximity to, the zone, the IMC 108 can facilitate presenting, viathe audio component 304 and/or the haptic component 306 of the interfacecomponent 106, the audible indicators and/or the haptic indicators tofacilitate notifying the user of the hazard.

In some implementations, the ARD 102 can obtain manuals (e.g., entiremanuals) relating to respective components and systems, wherein suchmanuals can be stored in the data store 114 or in an external datastore. The IMC 108 can make all or a desired portion of the manualsavailable for viewing on the display screen of the interface component106 by the user, for example, in response to a request or commandreceived from the user. The IMC 108 also can provide a search enginethat can allow the user to enter search requests, via the interfacecomponent 106, to search for desired information in a manual(s). Thesearch engine can perform the search and generate search results, basedat least in part on the search request, and can present, via the displayscreen, the search results to the user. The user can select a desiredsearch result, via the interface component 106, to view the informationfrom the manual(s) associated with the search result.

Users also can use the ARD 102 to create or build integration screens(e.g., electronic operator interfaces (EOIs)) for different (e.g.,custom) views and different users (e.g., operators) associated with thesystem. For instance, the IMC 108 can receive instructions and/or otherinformation from an ARD user via the interface component 106 (e.g.,using hand gestures, verbal information, and/or textual or controlinput). The instructions and/or other information can relate to controlsthat are to be employed on a screen, respective locations of thecontrols on the screen, respective regions for respective types of data,the respective types of data to be presented in the respective regionson the screen, and/or other features or parameters of the screen. TheIMC 108 can generate the screen based at least in part on the results ofanalyzing the instructions and/or other information. The IMC 108 canfacilitate presenting, via the display screen of the display component302, the screen (e.g., customized screen) to the user (e.g., in responseto a request for or selection of the screen, and/or dynamically orautomatically, in response to detected conditions associated with thesystem related to the screen).

In some implementations, the ARD 102, employing the IMC 108, cangenerate and display (e.g., the user can use the ARD 102 to generate anddisplay) ladder logic and screens as such ladder logic and screens arecreated or built. For instance, the IMC 108 can receive instructionsand/or other information from the user via the interface component 106(e.g., using hand gestures, verbal information, and/or textual orcontrol input). The instructions and/or other information can relate tovarious aspects (e.g., components, nodes, controls, desired data,desired routines, nets, I/O points, . . . ) relating to ladder logic ora screen. The IMC 108 can generate (e.g., create, build) the ladderlogic or the screen based at least in part on the results of analyzingthe instructions and/or other information. The IMC 108 can facilitatepresenting, via the display screen of the display component 302, theladder logic or the screen to the user when selected by the user or atother suitable times.

In connection with the building of ladder logic or a screen, the ARD102, via the IMC 108 and the display screen of the interface component,also can light up or make active affected nodes, nets, I/O points, orother elements, when they are affected (e.g., modified or set) orselected (e.g., by the user). This can facilitate letting the user knowthat a particular node, net, I/O point, or other element has beenaffected by an action performed by the user or by the ARD 102.

In some implementations, in connection with the building of ladder logicor a screen, the user also can use the ARD 102 for instruction lookup orlookup of other elements of ladder logic or a screen. For example, theuser can submit a request for instructions (e.g., ladder logicinstructions), screen elements, or other logic using natural languagevia the interface component 106 (e.g., by providing verbal inputinformation, textual input information, or other information). The IMC108 can analyze the request received in natural language, and employingnatural language recognition, the IMC 108 can interpret the naturallanguage of the request to determine a refined search request, based atleast in part on the natural language request.

The IMC 108 can employ a search engine to perform the refined search ofa database(s) (e.g., in the data store 114 or from another datasource(s)) that can comprise ladder logic instructions, screen elements,various other types of logic, or various other types of elements. Thesearch engine can generate and provide search results based at least inpart on the refined search. The IMC 108 can facilitate presenting, viathe interface component 106 (e.g., display screen), the search resultsto the user, wherein the search results can be recommended or suggestedladder logic instructions, screen elements, other logic, other elementsfor the user to consider in connection with building the ladder logic orscreen. The user can select one or more of the desired search resultsand can use the associated ladder logic instructions, screen elements,other logic, and/or other elements to facilitate building the ladderlogic or screen. The IMC 108 can facilitate generating the ladder logicor screen based at least in part on the search results and associatedinstructions or elements selected by the user. The IMC 108 canfacilitate presenting, via the display screen of the display component302, such ladder logic or the screen to the user, for example, whenselected by the user or at other suitable times. The ARD 102, byenabling the user to use the ARD 102 for instruction lookup or lookup ofother elements can enable or allow a natural language gateway toprecision logic encoding in connection with the design.

The ARD 102 can enable a user to ask (e.g., verbally or with eye (e.g.,sight or focus), facial, or hand gestures), via the interface component106, what a particular thing is (e.g., a wire, tag, I/O, component,voltage, configuration, element, . . . ) and ask for assistance or foradditional information, such as, for example, datasheets, applicationnotes, or schematics relating to the design of the system. The IMC 108can analyze the question, assistance request, and/or information requestpresented by the user. Based at least in part on the analysis results,the IMC 108 can determine a response to the question (e.g., the item youinquired about is a wire connected between device A and device B in thesystem design) and present the response to the user, can determine asuitable response to the assistance request and/or can provide orfacilitate providing the requested assistance, and/or can retrieve(e.g., from the data store 114 or other data source) and present therequested information to the user. The response to the question, theresponse to the assistance request and/or the requested assistance,and/or the additional information can be presented to the user as visualinformation via the display screen of the display component 302, and/oras audio information via the audio component 304.

In some embodiments, the ARD 102 also can employ a virtual eagle eyesfeature that can enable the user of the ARD 102 to magnify a portion ofthe field of view of the user as the user views the environment throughthe lens component (e.g., lenses) of the ARD 102. The magnification bythe ARD 102 can be enabled and controlled by the IMC 108, and can beimplemented via the image capture component 202 of the sensor component104, and the display screen of the display component 302 of theinterface component 106. The magnification of the virtual eagle eyesfeature implemented by the ARD 102 can be a magnifying glass, a virtualmagnifying glass, a microscope, or a virtual microscope.

The ARD 102, via use of the lens component of the ARD 102, the sensorcomponent 104 (e.g., the image capture component 202), and the interfacecomponent 106 (e.g., display component), can allow area magnification,for example, using eyeball focus of the user (e.g., on an object(s)) toplace or position the virtual magnifying glass. The sensor component104, using the user-facing camera(s) of the image capture component 202,can detect the focus of the eyes of the user as the user views throughthe lens component. The interface component 106 can receive instructionsfrom the user (e.g., textual instructions, verbal instruction, visualinstructions or gestures) via the interface component 106 and/or thesensor component 104 can detect conditions associated with the user(e.g., the facial expression of the user, the movement of the eyes oreye portion of the face), wherein the instructions or conditions canindicate that the user wants to or at least potentially wants to utilizethe virtual eagle eyes function. Based at least in part on the resultsof analyzing the instructions or conditions, the IMC 108 can determinewhether the virtual eagle eyes function is to be implemented and how(e.g., to what degree of magnification) it is to be implemented.

In accordance with various embodiments, the IMC 108 can control thesensor component 104 (e.g., the image capture component 202 of thesensor component 104) and the display component 302 of the interfacecomponent 106 to employ a boxed magnification version or a distortedview version of the object(s) in the environment that is to bemagnified. With regard to boxed magnification, the IMC 108 canfacilitate creating a boxed region (e.g., rectangular shaped, roundshaped, or irregular shaped region) over a portion of the area viewableby the user based at least in part on the eye focus of the user, asdetermined by sensor data from the sensor component 104 (e.g., theuser-facing camera(s) of the image capture component 202). The IMC 108can determine a level of magnification to employ to magnify at least aportion of the visual data (e.g., object(s) or portion of the object(s))in the boxed region, based at least in part on the results of theanalysis of the instructions, the sensor information, user preferences,and/or other information, in accordance with the defined informationmanagement criteria. The IMC 108, employing the interface component 106(e.g., the display screen of the display component 302), can facilitatemagnifying (e.g., presenting a magnified view of) at least the portionof the visual data in the boxed region, while the remaining visual dataoutside of the boxed region can be presented on the display screen ofthe ARD 102 at a non-magnified level or can be de-emphasized (e.g.,demagnified).

With regard to distorted view version of magnification, the IMC 108 canfacilitate creating a region (e.g., rectangular shaped, round shaped, orirregular shaped region) over a portion of the area viewable by the userbased at least in part on the eye focus of the user, as determined bysensor data from the sensor component 104 (e.g., the user-facingcamera(s) of the image capture component 202). The IMC 108 can determinea level of magnification to employ to magnify at least a portion of thevisual data (e.g., object(s) or portion of the object(s)) in the regionassociated with (e.g., corresponding to) the eye focus of the user,based at least in part on the results of the analysis of theinstructions, the sensor information, user preferences, and/or otherinformation, in accordance with the defined information managementcriteria. The IMC 108, employing the interface component 106 (e.g., thedisplay screen of the display component 302), can facilitate magnifying(e.g., presenting a magnified view of) at least the portion of thevisual data in the region, while the remaining visual data outside ofthe region can be presented on the display screen of the ARD 102 asbeing blurred, distorted, or otherwise de-emphasized.

The ARD 102, by employing such magnification techniques, can causeobjects within the gaze of the user to appear larger than life to theuser of the ARD 102 when the user is viewing the objects (e.g.,magnified objects) via the display screen of the interface component106. The IMC 108 of the ARD 102 also can generate and provide additionalmetadata, such as tags, or other information by voice or gesture commandreceived by the sensor component 104 and/or interface component 106 fromthe user to call for (e.g., to request) these additional layers ofinformation (e.g., datasheet, price and availability on maintenancestock shelves, virtual voltage probe, conformity to derating guidelines,etc.). The ARD 102 further can facilitate performing one or moreadditional actions, such as modifying (e.g., forcing or toggling) statesor bits associated with a system (e.g., a device of the system), forexample, in response to a voice or gesture command received by the ARD102 (e.g., the sensor component 104 or interface component 106) from theuser, wherein the voice or gesture command requests or instructs thatthe state or bit be modified.

In accordance with various implementations or embodiments, the ARD 102can be used to facilitate enhancing security (e.g., securityassistance), for example, by identification of operators, peers, andlocation, as well as security policy enforcement. It can sometimes bedesired (e.g., due to importance) to restrict access of users or devicesto certain devices, systems and functions based at least in part onsecurity credentials (e.g., user or operator security credentials,device security credentials). However, it may be difficult to robustlyassess user identity, and existing methods of determining identity canbe deficient, as, for example, passwords of a user can be shared withothers or hacked, keys can be duplicated, etc.

The ARD 102 can facilitate enhancing security associated with a systemby desirable identification of operators, peers, and location, as wellas security policy enforcement. The disclosed subject matter describesvarious modes of identifying a user for purposes of security (e.g.,security of the system, or a device or functions associated with thesystem). In some implementations, the ARD 102 can use multifactoridentification, employing a number of the capabilities of the ARD 102(e.g., the features and capabilities of sensors of the sensor component104 and/or the interfaces of the interface component 106), wherein thesensors, interfaces, and/or IMC 108 can interact with upstream databasesand computing power.

The ARD 102 can employ various security factors to facilitateidentifying or authenticating a user of the ARD 102. The varioussecurity factors that the ARD 102 can employ can comprise, forexample, 1) audible information, such as a spoken password, voicerecognition of a user's voice, audibly reading a human-only-readabletext, and/or answering security questions; 2) user-facing video toassess iris, retina, facial dimensions, and/or eye focus of the user; 3)outward facing video to enable the use of gestures of a user to enterpasswords, answer security questions, perform defined gestures (e.g.,which can be compared to the user's previously recorded motion andmannerism), fingerprints, hand, and/or other body dimensions of theuser; and/or 4) accelerometers and/or another sensor to recognize a gaitof a user.

With regard to the audible information, the IMC 108 can request that theuser present, via the audio sensor component 204, a spoken password, asample of the voice of the user (e.g., recite a particular phrase), areading of a human-only-readable text by the user, and/or an answer(s)to one or more security questions. When a spoken password is thesecurity factor, the IMC 108 can analyze the spoken password submittedby the user and can compare it to a stored password (e.g., spokenpassword), which can be retrieved from the data store 114 or anotherdata source (e.g., external data store), to determine whether the spokenpassword submitted by the user matches, or at least substantiallymatches, the stored password. In response to determining that the spokenpassword matches or substantially matches the stored password, the IMC108 can determine that the spoken password satisfies the definedsecurity criteria relating to spoken passwords. In response todetermining that the spoken password does not match or substantiallymatch the stored password, the IMC 108 can determine that the spokenpassword does not satisfy the defined security criteria relating tospoken passwords.

When a sample of the voice of a user is the security factor, the IMC 108and/or the recognition component 212 can analyze and employ voicerecognition techniques on the user's voice sample to identify voicecharacteristics of the user's voice. The IMC 108 can retrieve storedvoice characteristics or a stored voice sample from the data store oranother data source. In the case of a stored voice sample, the IMC 108and/or the recognition component 212 can analyze and employ voicerecognition techniques on the stored voice sample to identify voicecharacteristics of the stored voice sample. The IMC 108 can compare thevoice characteristics of the user's voice to the (stored) voicecharacteristics of the stored voice sample to determine whether thevoice characteristics of the user's voice matches or substantiallymatches the (stored) voice characteristics of the stored voice sample.In response to determining that the voice characteristics of the user'svoice matches or substantially matches the (stored) voicecharacteristics of the stored voice sample, the IMC 108 can determinethat the user's voice sample satisfies the defined security criteriarelating to voice recognition. In response to determining that the voicecharacteristics of the user's voice do not match or substantially matchthe (stored) voice characteristics of the stored voice sample, the IMC108 can determine that the user's voice sample does not satisfy thedefined security criteria relating to voice recognition.

When a reading of a human-only-readable text by the user is the securityfactor, the IMC 108 can analyze the reading of the human-only-readabletext submitted by the user and can compare it to a storedhuman-only-readable text (e.g. in text or audible form), which can beretrieved from the data store 114 or another data source (e.g., externaldata store), to determine whether the human-only-readable text submittedby the user matches, or at least substantially matches, the storedhuman-only-readable text. In response to determining that the submittedhuman-only-readable text matches or substantially matches the storedhuman-only-readable text, the IMC 108 can determine that the submittedhuman-only-readable text satisfies the defined security criteriarelating to human-only-readable text. In response to determining thatthe submitted human-only-readable text does not match or substantiallymatch the stored human-only-readable text, the IMC 108 can determinethat the submitted human-only-readable text does not satisfy the definedsecurity criteria relating to human-only-readable text.

When the answering of a security question is the security factor, theIMC 108 can analyze the security question answer submitted by the userand can compare it to a stored security question answer (e.g., spoken ortextual security question answer), which can be retrieved from the datastore 114 or another data source (e.g., external data store), todetermine whether the security question answer submitted by the usermatches, or at least substantially matches, the stored security questionanswer. In response to determining that the submitted security questionanswer matches or substantially matches, the stored security questionanswer, the IMC 108 can determine that the submitted security questionanswer satisfies the defined security criteria relating to securityquestions. In response to determining that the submitted securityquestion answer does not match or substantially match the storedsecurity question answer, the IMC 108 can determine that the submittedsecurity question answer does not satisfy the defined security criteriarelating to security questions.

With regard to using user-facing video to assess iris, retina, facialdimensions, and/or eye focus of the user as a security factor, the IMC108 and/or the recognition component 212 can analyze and employbiometric recognition techniques on the user's biometric information(e.g., iris, retina, facial dimensions, and/or eye focus of the user) toidentify characteristics of the user's biometric information. The IMC108 can retrieve stored characteristics or stored biometric informationfrom the data store or another data source. In the case of storedbiometric information, the IMC 108 and/or the recognition component 212can analyze and employ biometric recognition techniques on the storedbiometric information to identify characteristics of the storedbiometric information. The IMC 108 can compare the characteristics ofthe user's biometric information to the (stored) characteristics of thestored biometric information to determine whether the characteristics ofthe user's biometric information matches or substantially matches the(stored) characteristics of the stored biometric information. Inresponse to determining that the characteristics of the user's biometricinformation matches or substantially matches the (stored)characteristics of the stored biometric information, the IMC 108 candetermine that the user's biometric information satisfies the definedsecurity criteria relating to biometric recognition. In response todetermining that the characteristics of the user's biometric informationdo not match or substantially match the (stored) characteristics of thestored biometric information, the IMC 108 can determine that the user'sbiometric information does not satisfy the defined security criteriarelating to biometric recognition.

With regard to using outward-facing video to assess gestures orbiometric features of the user as a security factor, the IMC 108 canrequest that the user present, via the image capture component 202,gestures to enter a password or answer security questions, performdefined gestures, and/or submit biometric information. When a passwordis the security factor, the IMC 108 can analyze the password submittedvia a hand gesture(s) of the user and can compare it to a storedpassword, which can be retrieved from the data store 114 or another datasource, to determine whether the submitted password matches, or at leastsubstantially matches, the stored password. In response to determiningthat the submitted password matches or substantially matches the storedpassword, the IMC 108 can determine that the submitted passwordsatisfies the defined security criteria relating to passwords. Inresponse to determining that the submitted password does not match orsubstantially match the stored password, the IMC 108 can determine thatthe submitted password does not satisfy the defined security criteriarelating to passwords.

When answering of a security question is the security factor, the IMC108 can analyze the security question answer submitted by the user usinga gesture(s) and can compare it to a stored security question answer,which can be retrieved from the data store 114 or another data source,to determine whether the security question answer submitted by the usermatches, or at least substantially matches, the stored security questionanswer. In response to determining that the submitted security questionanswer matches or substantially matches the stored security questionanswer, the IMC 108 can determine that the submitted security questionanswer satisfies the defined security criteria relating to securityquestions. In response to determining that the submitted securityquestion answer does not match or substantially match the storedsecurity question answer, the IMC 108 can determine that the submittedsecurity question answer does not satisfy the defined security criteriarelating to security questions.

With regard to using performed defined gestures as a security factor,the IMC 108 can analyze defined gestures performed by the user and cancompare the performed defined gestures to stored defined gestures, whichthe IMC 108 can retrieve from the data store 114 or another data source.The stored defined gestures associated with the user can comprise, forexample, previously performed defined gestures of the user that havebeen recorded. The motions and mannerisms of the user in performing thedefined gestures can be assessed along with the gestures themselves. Thegestures can involve the user using one or more of the user's finger(s),hand(s), and/or other body parts in particular ways (e.g., to makeparticular gestures, motions, or mannerisms). In response to determiningthat the performed defined gestures match or substantially match thestored defined gestures, the IMC 108 can determine that the performeddefined gestures satisfy the defined security criteria relating toperformed defined gestures. In response to determining that theperformed defined gestures do not match or substantially match thestored defined gestures, the IMC 108 can determine that the performeddefined gestures do not satisfy the defined security criteria relatingto performed defined gestures.

With regard to using biometric information of the user as a securityfactor, the IMC 108 and/or the recognition component 212 can analyze andemploy biometric recognition techniques on the user's biometricinformation (e.g., fingerprint(s), hand, and/or body dimensions of theuser) to identify characteristics of the user's biometric information,and/or on stored biometric information to identify characteristics ofthe stored biometric information (e.g., if characteristics of the storedbiometric information have not already been determined). The IMC 108 cancompare the characteristics of the user's biometric information to the(stored) characteristics of the stored biometric information todetermine whether the characteristics of the user's biometricinformation matches or substantially matches the (stored)characteristics of the stored biometric information. In response todetermining that the characteristics of the user's biometric informationmatches or substantially matches the (stored) characteristics of thestored biometric information, the IMC 108 can determine that the user'sbiometric information satisfies the defined security criteria relatingto biometric recognition. In response to determining that thecharacteristics of the user's biometric information do not match orsubstantially match the (stored) characteristics of the stored biometricinformation, the IMC 108 can determine that the user's biometricinformation does not satisfy the defined security criteria relating tobiometric recognition.

With regard to using the gait of the user as a security factor, the IMC108 can analyze sensor data relating to the gait of the user todetermine characteristics of the gait of the user, wherein the sensordata can be obtained from an accelerometer or other sensor of the sensorcomponent 104. The IMC 108 can compare the characteristics of the gaitof the user to stored characteristics of the gait of the user, which canbe retrieved from the data store 114 or another data source. The storedcharacteristics of the gait of the user can be determined by analyzingprevious sensor data relating to the gait of the user. The IMC 108 cancompare the characteristics of the gait of the user to the storedcharacteristics of the gait of the user. In response to determining thatthe characteristics of the user's gait matches or substantially matchesthe stored characteristics of the user's gait, the IMC 108 can determinethat the user's gait satisfies the defined security criteria relating togait recognition. In response to determining that the characteristics ofthe user's gait do not match or substantially match the storedcharacteristics of the user's gait, the IMC 108 can determine that theuser's gait does not satisfy the defined security criteria relating togait recognition.

In accordance with various embodiments, the ARD 102 can employ one, morethan one, or all of those security factors (or another securityfactor(s)) to facilitate identifying or authenticating a user of the ARD102. For example, the IMC 108 of the ARD 102 can request that a userpresent audible information in the form of a spoken password to theaudio sensor component 204, and provide biometric information in theform of retina characteristics, as detected by the image capturecomponent 202. The IMC 108 can determine whether the user issufficiently identified or authenticated, based at least in part on theresults of analyzing those two security factors (e.g., spoken passwordand retina assessment).

As another example, the IMC 108 of the ARD 102 can request that a userpresent audible information in the form of a human-only-readable text tothe audio sensor component 204, provide biometric information in theform of a fingerprint, as detected by the image capture component 202,and perform defined gestures. The IMC 108 can determine whether the useris sufficiently identified or authenticated, based at least in part onthe results of analyzing those three security factors (e.g.,human-only-readable text, fingerprint, and perform defined gestures).

In some embodiments, the IMC 108 can randomly select the number ofsecurity factors to employ, and/or the types of security factors toemploy, to identify or authenticate a user of the ARD 102. In someimplementations, the IMC 108 can utilize a random number generator(e.g., real or pseudo random number generator) to generate a randomvalue(s) and/or a randomizing function(s), wherein the random value(s)and/or randomizing function(s) can be used by the IMC 108 to randomizethe selection of the number of security factors to employ, and/or thetypes of security factors to employ, to identify or authenticate a userof the ARD 102. In other implementations, an authorized person candetermine and the select the number and/or type of security factors toemploy to identify or authenticate a user of the ARD 102.

For example, on a first occasion, utilizing the random number(s) and/orrandomizing function(s), the IMC 108 can randomly determine that twosecurity factors are to be employed, and can randomly select a spokenpassword and a fingerprint scan as the two types of security factors tobe employed to identify or authenticate a user of the ARD 102. On asecond occasion, utilizing the random number(s) and/or randomizingfunction(s), the IMC 108 can randomly determine that three securityfactors are to be employed, and can randomly select answering a securityquestion, a retina scan, and a password entered via gestures of the useras the three types of security factors to be employed to identify orauthenticate a user of the ARD 102.

Employing randomization in determining the number and/or type ofsecurity factors to be employed to identify or authenticate a user canenable the security process to be relatively fast and flexible. Therandom and/or multi-factor security process can be flexible orapproximate in some areas, which can allow variability (e.g., allow fordaily, periodic, or aperiodic variability with regard to the numberand/or type of security factors used), as desired (e.g., by theauthorized person). The flexibility or approximation of the securityprocess can be balanced with the multi-security factor process, as theemployment of multiple and/or random security factors in the securityprocess can compensate for any approximation in some areas associatedwith the security process. The security process, by employing randomizedsampling and selection of multiple security factors, can provide robustsecurity for the ARD 102 and associated information, devices, and/orsystems and can make it relatively difficult for imposters (e.g.,unauthorized or malicious persons or entities) to fake an identity ofanother person (e.g., an authorized user).

If the user does not pass the security check (e.g., authentication), theIMC 108 can deny the user access rights to the ARD 102 and/or associateddevices or systems (e.g., industrial automation system), or can grantlimited access rights to the ARD 102 and/or associated devices orsystems (e.g., limited rights that can be granted to non-authenticatedusers or guests). If the user (e.g., operator, or supervisor) does passthe security check (e.g., authentication), the IMC 108 can grant a setof access rights to the user to access and use the ARD 102 and/orassociated devices or systems, in accordance with the set of accessrights granted to the user, the defined information management criteria,and/or the defined security criteria. The IMC 108 can control access tothe ARD 102, information, activities, screens (e.g., HMIs, EOIs, . . .), devices, processes, assets, changes (e.g., changes made to data;changes made to machine, HMI, or system design; changes made toparameters), virtual penetration (e.g., inside the box view and action),and/or force/run manipulation, etc., based at least in part on the setof access rights granted to the user.

The set of access rights granted to the user can be based at least inpart on the role (e.g., operator, or supervisor) of the user, securityclearance of the user, the location of the user (e.g., in relation tothe system; is user in a remote location away from the system or is theuser in the same location as the system), and/or other desired factorsrelating to access rights, in accordance with the defined informationmanagement criteria and/or defined security criteria (e.g., as specifiedby the defined security policy). The set of access rights can indicateor specify which information, activities, screens, devices, processes,assets, changes, virtual penetration, and/or force/run manipulation,etc., that the user (e.g., authenticated and authorized user) ispermitted to access, use, or perform, accordingly.

In accordance with various embodiments, the ARD 102 can employ securityscan and patrol by checking (e.g., using out-facing cameras of the imagecapture component 202) other personnel in the area. For example, the ARD102 can analyze video, images (e.g., photographs), and/or audio capturedby the ARD 102 (e.g., image capture component 202, audio sensorcomponent 204). Based at least in part on the analysis results, the ARD102 can determine who other persons near the user of the ARD 102 are,determine whether such other persons have authorization to be in thearea where the ARD 102 is currently located and at the stations (e.g.,work stations) where such other persons are located. The ARD 102 (e.g.,IMC 108 of the ARD 102) also can determine information regarding suchother persons (e.g., name, years of experience, security level, etc., ofthe other person), wherein the ARD 102 can present that information(e.g., visual information, or audio information) to the user (e.g.,goggle wearer) wearing the ARD 102.

For instance, the ARD 102 can use out-facing cameras of the imagecapture component 202 and/or other sensors (e.g., audio sensor component204) of the sensor component 104 to facilitate identifying otherpersonnel (e.g., workers) that are within the field of view of theout-facing cameras or within the field of view of any cameras (evenfacing behind the user) of the image capture component 202 by and basedat least in part on any number and/or type of the security factors(e.g., audible information, such as spoken password, voice recognitionof other person's voice, other person audibly reading ahuman-only-readable text, and/or other person answering securityquestions; assess, iris, retina, facial dimensions, and/or eye focus ofthe other person; use of gestures of other person to enter passwords,answer security questions, performed defined gestures, fingerprints,hand, or other body dimensions of the other person; and/or recognizegait of the other person), as disclosed herein.

For example, the ARD 102 can employ (e.g., randomly select) performingfacial recognition on the other person and requesting the other personprovide a spoken password as the security factors for identifying orauthenticating the other person. The user of the ARD 102 can employ theoutward-facing camera(s) of the image capture component 202 to capturean image of the face (or portion thereof) of another person (e.g., ARDuser or other person not using an ARD) in proximity to the user (e.g.,within the field of view of the user or outside the field of view of(e.g., behind) the user). The IMC 108 and/or recognition component 212can employ facial recognition techniques to perform facial recognitionon the image of the face (or portion thereof) of the other person toidentify or authenticate, or attempt to identify or authenticate, theother person. The IMC 108 and/or user can request the other person toprovide a valid spoken password, and the audio sensor component 204(e.g., microphone) can sense or capture the spoken password provided bythe other user. The IMC 108 can compare the spoken password submitted bythe other person to a stored password to identify or authenticate, orattempt to identify or authenticate, the other person.

Based at least in part on the identification or authentication of otherpersons (e.g., in proximity to the user of the ARD 102), the IMC 108 ofthe ARD 102 can determine whether any or all of the other persons areauthorized to be working at the station at which they are located. Forexample, with regard to the other user, if the IMC 108 determines thatthe other person is identified and/or authenticated, based at least inpart on the results of performing facial recognition and analyzing thespoken password submitted by the other user, the IMC 108 can determinewhether the other person is authorized to be in the area (e.g.,particular work station in the area) in which the person is locatedbased at least in part on the set of access rights associated with theother person and the defined security criteria. For instance, the IMC108 can determine the location of the other user (and/or determine thework station at which the other person is located) based at least inpart on the results of analyzing sensor data relating to the location ofthe other user and obtained from the sensor component 104. The IMC 108can analyze the set of access rights associated with the other person todetermine the locations (and/or work stations) that the other person isauthorized to access, and can determine whether the other person isauthorized to be at the location (and/or work station) the other personis currently located, based at least in part on the results of comparingthe current area (and/or work station) where the other person is locatedto the locations (and/or work stations that the other person isauthorized to access.

If the IMC 108 determines that the other person is authorized to be atthe current location (and/or work station), no further action is needed.If the IMC 108 determines that the other person is not authorized to beat the current location (and/or work station), the IMC 108 or the usercan take action (e.g., call security personnel, transmit notificationand/or identifying information (e.g., image, such as a photograph, ofthe other person) or other information (e.g., location of the otheruser) to device (e.g., ARD) of security personnel, request that theother person leave the area or work station) to address the issue of anunauthorized person being in that location (and/or work station).

In response to identification or authentication of the other person(s)by the ARD 102, the ARD 102 also can provide feedback to the user of theARD 102 about the other person(s). For example, with regard to the otherperson, in response to identification or authentication of the otherperson by the ARD 102, the IMC 108 can generate customized information,comprising augmented reality information, relating to the other person.The IMC 108 can obtain information relating to the other user from thedata store 114 and/or another data source(s) (e.g., a database(s)associated with the company), and can process such information togenerate the customized information. The IMC 108 can facilitatepresenting, via the display screen of the interface component 106 (orvia the audio component 304), the customized information, or desiredportion thereof, to the user to provide the user additional informationregarding the other person. The customized information relating to theother person can comprise, for example, the name of the other person,the years of employment experience or resume of the other person,security level granted to the other person, job function of the otherperson, the frequency or history of the user meeting with the otherperson (e.g., history of relationship between the user and otherperson), and/or other pertinent information or details that canfacilitate improving security and efficacy of the personnel (e.g., teamof users) and the system. This also can allow human augmentation ofsecurity aspects associated with the system.

The security policy enforcement employed or implemented by the ARD 102(e.g., employing the defined information management criteria and thedefined security criteria), can provide advantages relating toindustrial activities including maintenance, design, safety, etc.,associated with the industrial automation system(s). Such securitypolicy enforcement can be implemented and can enhance other aspects toimprove the industrial protection of processes. For example, thesecurity policy enforcement employed or implemented by the ARD 102 canfacilitate improving the design environment associated with a system byproviding a secure development environment whose virtues can be extolledelsewhere. The security policy enforcement employed or implemented bythe ARD 102 also can facilitate improving the assembly process to assurethat all parts are assembled by legitimate and authorized users (e.g.,operators). Safety often can be intertwined with security, and thesecurity policy enforcement employed or implemented by the ARD 102 canfacilitate improving safety with regard to the system and usersassociated therewith.

The ARD 102 also can be employed to enhance safety of people, industrialmachines, systems, and processes, and machine and HMI design activities(e.g., the ARD can provide safety assistance). Problems sometimes canarise that can impair safety in an industrial environment. Such problemscan include or relate to, for example, the inability to adequatelydetermine hazards and hazardous zones, inability to call for help easilyin some situations, navigating in a visibility restricted area,reviewing incidents, and/or controlling machines from a safe distance.

The disclosed subject matter, employing the ARD 102, can overcome oraddress these and/or other issues relating to safety and hazards. Inaccordance with various embodiments, with the user (e.g., operator)interacting with or wearing the ARD 102, the ARD 102 can assist the userin a number of ways, such as disclosed herein, to improve speed of theuser with regard to performing work tasks (e.g., maintenance, assembly,or security) tasks, quality of work performed, security of theperformance of work tasks, and safety of the user and the environment inthe performance of work tasks. The ARD 102 can advise or instruct theuser through visual signals, audible signals, and/or haptic feedback tofacilitate enhancing the safety associated with the system and the user,and assisting in the performance (e.g., safe performance) of work tasksassociated with the system (e.g., enhancing the safety associated withthe performance of work tasks associated with the system).

The ARD 102 can be employed to overcome issues relating to an inabilityor inadequate ability to determine hazards and hazardous zones in anarea (e.g., an outdoor area, a building, a plant, . . . ). In someimplementations, a designer, using the ARD 102 (or another device), canencode information relating to hazardous components and/or hazardousregions, and the safe zones (e.g., keepout zones, arc flash zones, . . .) that can surround users in an area(s) associated with a system (e.g.,industrial automation system). For instance, a user (e.g., designer)and/or the IMC 108 of the ARD 102 can identify or determine therespective locations of hazardous components and/or hazardous regions inan area, for example, by, based at least in part on sensor data relatingto the conditions (e.g., environment conditions) associated with suchhazardous components and/or hazardous regions, respective locations ofsuch hazardous components and/or hazardous regions, and/or other data(e.g., specifications, hazard or warning information, or otherinformation regarding components) obtained from another source (e.g.,data store 114 or an external database(s)). The sensor data can indicatethe respective locations, orientations, and/or movements of respectivecomponents (e.g., hazardous components) and/or regions (e.g., hazardousregions. The sensor data also can indicate hazardous conditions (e.g.,hot temperature, high voltage, high current, chemical hazard, explosivehazard, . . . ) associated with components and/or regions in the area.

Based at least in part on the results of analyzing the sensor dataand/or the other data, the IMC 108 can determine one or more hazardouscomponents and/or hazardous regions in the area, and the respectivelocations of the one or more hazardous components and/or hazardousregions. The IMC 108 can be used (e.g., by the user) to encodeinformation regarding the one or more hazardous components and/orhazardous regions. The encoded information regarding a hazardouscomponent or hazardous region can comprise, for example, the type ofhazard, the hazard level of the hazard, the location of the hazardouscomponent or hazardous area, a component identifier of the componentassociated with the hazard, and/or other desired information. The IMC108 can store the encoded information regarding the one or morehazardous components or hazardous regions in the data store 114 of theARD 102 and/or in an external data store, and/or can provide the encodedinformation to other ARDs.

Also, based at least in part on the results of analyzing the sensor dataand/or the other data, the IMC 108 can identify or determine one or moresafe zones (e.g., keepout zones, arc flash zones, . . . ) that cansurround the one or more hazardous components or hazardous regions inthe area, and the respective locations of the one or more safe zones.The IMC 108 can be used (e.g., by the user) to encode informationregarding the one or more safe zones. The encoded information regardinga safe zone can comprise, for example, an indication or identifier thatthe zone is a safe zone, the safety level of the safe zone, the locationof the safe zone, and/or other desired information. The IMC 108 canstore the encoded information regarding the one or more safe zones inthe data store 114 of the ARD 102 and/or in an external data store,and/or can provide this encoded information to other ARDs.

As the user of the ARD 102 uses, wears, or interacts with the ARD 102 toview the environment in the location where the user is located and/ormoves through the area, the IMC 108 of the ARD 102 can determine thelocation, orientation, and/or motion (e.g., movement) of the user, andthe user's field of view through the lens component of the ARD 102,based at least in part on the sensor data relating to location,orientation, and/or motion of the user that is received from the sensorsof the sensor component 104, as disclosed herein. The IMC 108 also canaccess the encoded information from the data store 114 or other datasource (e.g., external data store) and analyze the encoded informationin view of the location, orientation, motion, and field of view of theuser.

Based at least in part on the results of analyzing the encodedinformation, the IMC 108 can determine whether there is a hazardouscomponent, hazardous region, or safe zone (e.g., associated with thearea and associated system) in the field of view of, or in proximity to,the user. In response to determining that there is a hazardouscomponent, hazardous region, or safe zone in the field of view of, or inproximity to, the user, the IMC 108 can generate customized information,comprising augmented reality information and/or virtual realityinformation, relating to the hazardous component, hazardous region,and/or safe zone. The customized information can include visual and/oraudio information (e.g., hazard indicator and/or other hazard-relatedinformation; safe zone indicator or information), and/or haptic feedback(e.g., vibratory signals or other haptic signals).

The hazard indicator (e.g., hazard warning indicator) or otherhazard-related information can indicate or specify the type of hazard(e.g., high temperature, high voltage level, high current level,chemical hazard, or explosion hazard, . . . ), the location (and totalregion) where the hazard exists or is a threat, the hazard level (e.g.,highly hazardous, moderately hazardous; hazard level is 5, . . . hazardlevel is 8, hazard level is 9, . . . ), parameter values related to thehazard (e.g., temperature is 300 degrees) and/or other desired data. Thesafe zone indicator or other safe zone related information can indicateor specify the type of safe zone (e.g., (e.g., keepout zone, arc flashzone, . . . ), the location (and total region) where the safe zoneexists, the safety level(s) (e.g., moderately safe, very safe; safetylevel is 1, safety level is 2, . . . ), parameter values related to thesafe zone, and/or other desired data.

The IMC 108 can facilitate presenting, via the display screen of thedisplay component 302, audio component 304, and/or the haptic component306, the customized information to the user. The user can perceive thecustomized information presented by the ARD 102 to the user and canthereby discover the hazardous component(s) and/or hazardous region(s),as well as the safe zone(s), in the field of view of, or in proximityto, the user, and pertinent information regarding the hazardouscomponent(s), hazardous region(s), and/or safe zone(s). The customizedinformation (e.g., augmented and/or virtual reality information)presented to the user by the ARD 102 can enable the user to safelynavigate around or otherwise avoid such hazards.

The customized information can include, for example, a visual hazardindicator and/or visual object (e.g., virtual object) that can be aparticular color (e.g., red) or color cast (e.g., a correspondinglycolored color cast) that can be presented on the display screen (e.g.,overlaid on the user's field of view) over the hazardous component orhazardous region as the user views the hazardous component or hazardousregion in the user's field of view. The visual hazard indicator and/orvisual object also can include lines or other markers (e.g.,first-down-markers) that can be associated with the color or color cast,wherein, for example, the lines can be spheres or concentric linessurrounding, and extending outward from, the location of a hazard. Insome implementations, the visual hazard indicator and/or visual objectcan be blinking on the display screen or can be solidly displayed on thedisplay screen. The lines or markers can indicate hazard perimeters(e.g., arc or spark perimeters, perimeters based on electrical safe workpractices, perimeters based on occupational safety and healthadministration (OSHA) standards or requirements, . . . ) determinedbased at least in part on design (e.g., system design), rules, and/oractive or live measurements of voltage and temperature associated withthe hazardous component or region (e.g., as sensed or measured by thesensor component 104). Other types of visible indications of thehazardous component or hazardous region can be employed by the ARD 102as well.

In certain implementations, when the user is initially detected to be inproximity to a hazardous component or region, the IMC 108 can facilitatepresenting a visual hazard indicator or visual object as blinking on thedisplay screen to begin warning the user of the hazard. If the usercontinues to get closer to the hazardous component or region, or reachesout to touch the hazardous component, or enters the hazardous region,the IMC 108 can facilitate presenting a solid visual hazard indicator orvisual object on the display screen to facilitate warning the user thatthe risk posed by the hazard to the user is increasing. As desired,alternatively, a solid hazard indicator or object can be the initialhazard warning, and a blinking hazard indicator or object can indicateto the user that the risk posed by the hazard to the user is increasing.

With further regard to safe zones, the customized information cancomprise, for example, a visual safe zone indicator and/or visual safezone object (e.g., virtual object) that can be a particular color (e.g.,yellow or green) or color cast (e.g., correspondingly colored colorcast) that can be presented on the display screen (e.g., overlaid on theuser's field of view) over the portion(s) of the area in the user'sfield of view where the safe zone(s) exist(s) as the user views the safezone(s) in the user's field of view. In some implementations, the visualsafe zone indicator and/or visual safe zone object can be blinking onthe display screen or can be solidly displayed on the display screenassociated with the lens component of the ARD 102 and the field of viewof the user. Other types of visible indications of the safe zone can beemployed by the ARD 102 as well.

As another example of a hazard warning, when the user of the ARD 102approaches or enters a hazardous region bodily (e.g., hand, foot, orwhole body) or with a tool, the IMC 108 can detect the location of theuser in relation to the hazardous region (e.g., can detect that the useris approaching or has entered the hazardous region with a part of hisbody or a tool), based at least in part on the results of analyzingsensor data relating to the location, orientation, and/or motion of theuser, sensor data relating to the location of the hazardous region,and/or sensor data relating to the user's body and/or associated tool(e.g., images of the body of user from the image capture component 202).In response to such detection, the IMC 108 can generate and facilitatepresenting a warning or notification to the user visually (e.g., via thedisplay screen of the display component 302), audibly (e.g., via theaudio component 304), and/or haptically (e.g., via the haptic component306) to warn or notify the user that the user is approaching or hasentered the hazardous region.

It is to be appreciated and understood that, as also disclosed herein,with regard to another hazardous component or region for which there isno encoded information (e.g., a new hazard), the IMC 108 can identify ordetermine the other hazardous component or region based at least in parton sensor data obtained from sensors of the sensor component 104 and/orother data. In response, the IMC 108 can generate and facilitatepresenting, to the user, via the display screen, audio component 304,and/or haptic component 306, customized information, comprisingaugmented and/or virtual reality information, relating to the hazardouscondition or region to facilitate warning and informing the user of thehazardous condition or region, as more fully described herein.

The ARD 102 or a user (e.g., system designer) also can determinehazardous components or hazardous regions using other encodedinformation regarding voltages, temperatures, or other conditionsassociated with the system. The IMC 108 or the user can determinetemperatures associated with components or regions in the area (e.g.,associated with the system) based at least in part on sensor datarelating to temperatures obtained from one or more sensors (e.g.,temperature sensor, such as an IR temperature sensor) of the sensorcomponent 104 or a camera(s) (e.g., out-facing camera(s), such as an IRcamera(s)) of the image capture component 202, and/or other dataobtained from another data source(s) (e.g., other sensor data obtainedfrom system sensors associated with the system; and/or other data storedin an external database). Similarly, the IMC 108 or the user candetermine voltages associated with components or regions in the areabased at least in part on sensor data relating to voltages obtained fromone or more sensors (e.g., voltage sensor) of the sensor component 104and/or system sensors (e.g., temperature sensors) associated with thesystem, and/or from other data obtained from another data source(s)(e.g., other data stored in an external database).

Knowing the hazards in an area, the IMC 108 or the user (e.g., systemdesigner) can determine or calculate hazardous regions surroundinghazards (e.g., hazardous components or other hazards based at least inpart on industry criteria (e.g., OSHA criteria, standards, orrequirements, Underwriters Laboratories (UL) criteria, standards, orrequirements, . . . ) and/or corporate criteria (e.g., corporatestandards). The IMC 108 of the ARD 102 or the user can create or build(e.g., dynamically create or build) the hazardous regions, which can be,for example, virtual versions (e.g., 3-D virtual versions) of the actualhazardous regions in the area associated with the system.

With regard to the determined or calculated hazardous regionssurrounding hazards, in response to detecting that the user andassociated ARD 102 is located in proximity to, or is viewing through theARD 102, a hazardous region in an area, the IMC 108 can facilitatepresenting, via the display screen of the interface component 106, thehazardous region (e.g., a 3-D virtual version of the hazardous region),and/or information (e.g., hazard-related information or other pertinentinformation), an alert, or a notification regarding the hazardous regionto provide the user relevant information regarding the hazardous region.As the user is viewing a portion of the area associated with the systemwherein the hazardous region is located through the lens component ofthe ARD 102, the IMC 108 can facilitate presenting, via the displayscreen associated with the lens component, the 3-D virtual version ofthe hazardous region such that the 3-D virtual version of the hazardousregion is overlaid in the user's field of view to appear over theposition of the actual hazardous region in the user's field of view. TheIMC 108 also can facilitate presenting other customized informationrelating to the hazardous region to the user to notify or warn the userof the hazardous region and its location, wherein the customizedinformation can comprise audio information presented by the audiocomponent 304 and/or haptic feedback presented by the haptic component306.

The ARD 102 also can be employed to address or overcome issues relatingto an inability of a person to call for help easily in certainsituations. Some hazards associated with a system can cause users (e.g.,operators) to be restricted in their means of calling for help if aproblem or injury occurs.

For instance, if a user of an ARD 102 is injured, the user may or maynot be able to use the ARD 102 or other device to call for help and noother person may be around to hear the user's call for help. If the useris able to call for help, the user can use the ARD 102 or other deviceto make a call or send a message for help (e.g., due to an injury to theuser or for another need or desire of the user). The ARD 102 can beassociated with a communication network, as more fully described herein,and the ARD 102 can communicate (e.g., transmit, broadcast) a help orassistance message or call of the user to other devices associated withother users (e.g., listeners) anywhere within the plant (e.g.,industrial plant comprising the system), the state, the country, or theplanet.

In certain instances, the user of the ARD 102 also may desire to make acall or send a message to another device (e.g., ARD or other device) torequest assistance or information, for example, to facilitate performingan activity or task with remote assistance (e.g., remote expertassistance) from another person. In such instances, the user can use theARD 102 to communicate, via call or message, to the other device throughthe communication network.

In some instances though, if the user is injured or incapacitated in away that the user cannot readily call for help, the ARD 102 can beemployed to detect (e.g., automatically detect) that the user isinjured, unconscious, or otherwise facing a problem for which assistancefor the user is desired (e.g., necessary), and can communicate (e.g.,automatically communicate) a message or call to other devices and/orpersons. For instance, the sensors (e.g., image capture component 202)of the sensor component 104 can sense gestures (e.g., hand, eye, orfacial gestures) that can indicate that the user is injured and/ordesires or requests assistance; the sensors (e.g., gyroscope oraccelerometer) can sense the orientation, acceleration, and/or motion ofthe user (e.g., to detect that the user is falling or has fallen down,and/or has bumped into an object); the sensors can detect a pulse rateof the user, wherein, for example, a very low pulse rate or a very highpulse rate can indicate that the user is facing a health problem and candesire assistance; the sensors can sense whether the user is consciousor not conscious (e.g., the user-facing camera of the image capturecomponent 202 can sense eye movement or focus (if any) of the user; theaccelerometer or gyroscope can sense whether the user is moving at all);and/or the sensors (e.g., audio sensor component 204) can detect verbalcommunications from the user or other sounds in proximity to the user,wherein the verbal communications of the user can indicate or specifythat the user desires or at least may desire assistance.

The IMC 108 can analyze the sensor data relating to the conditions ofthe user obtained from the sensors of the sensor component 104 and/orother data. Based at least in part on the results of the analysis, theIMC 108 can determine whether assistance is to be provided to the userand the location of the user. The IMC 108 can communicate (e.g.,transmit, broadcast) a request for assistance message or emergencyassistance message, and/or an associated alert, to one or more otherdevices (e.g., ARDs, communication devices) associated with thecommunication network to facilitate notifying other users or entitiesthat the user is or may be in need of assistance, the location of theuser, and/or other details relating to the condition of the user (e.g.,detected heart rate of user, detection of substance that may be blood inproximity to the user, detection that user is unconscious). If therequest for assistance message or emergency assistance message, and/orassociated alert, is broadcasted by the ARD 102, a number of ARDs orother communication devices can receive such message(s) and/orassociated alert. If the request for assistance message or emergencyassistance message, and/or associated alert, is transmitted by the ARD102 to a particular device, such device can receive the message and/oralert.

In some embodiments, in response to determining that a request forassistance (e.g., emergency assistance) is to be sent, the IMC 108 canemploy automatic arc-flash detection to wirelessly transmit a distresscall or message to another device or entity to indicate that the user oranother person desires (e.g., needs) assistance.

Another issue that can arise can relate to a user (e.g., operator orother employee, emergency person, or first responder) having to navigatein a visibility restricted area. The disclosed subject matter, employingthe ARD 102, can enable a user to navigate in a visibility restrictedarea and/or in an area where hazards may not be visible or readilyvisible.

The ARD 102 can employ the sensor component 104 to obtain (e.g., detect)location information and/or orientation information relating to thelocation and/or orientation of the user to facilitate determining alocation and/or orientation of the user, as more fully described herein.The IMC 108 of the ARD 102 also can determine the layout of an area(e.g., a building, an industrial automation system, structure, anoutdoor area), including the relative and respective locations ofobjects and/or other features in the area, based at least in part onsensor data obtained from sensors of the sensor component 104 and/orother data from another data source(s) (e.g., data store 114 or externaldata store). For instance, sensors (e.g., camera and/or IR camera of theimage capture component 202, recognition component 212, environmentsensors, . . . ) of the sensor component can sense and detect the layoutof the area, including objects and other features of the area, and therelative and respective locations of the objects and other features inthe area to each other. The objects and other features can include, forexample, walls, doorways, doors, furniture, barriers, machines, devices,assets (e.g., industrial assets), appliances, stairs, building fixtures,lighting fixtures, wires (e.g., electrical wires, cable wires, . . . ),railings, rails, logs, tracks, trees, plants, holes, ditches, wells,fences, hills, curbs, precipices, vehicles, and/or any other type ofobject or features having a defined physical presence or shape. Thesensors of the sensor component 104 can generate sensor data relating tothe objects and other features of the area. The other data can comprise,for example, plans, blueprints, or layout information for a building,other structure, or system (e.g., industrial automation system), map(s)of the area, and/or aerial or satellite images.

The IMC 108 can analyze the sensor data and/or the other data, and candetermine the layout of the area, including the relative and respectivelocations of objects and/or other features in the area, based at leastin part on the sensor data and/or the other data. The IMC 108 also candetect one or more hazardous conditions or regions (e.g., hightemperature, high voltage, high current, fire, smoke, chemical hazard,explosive hazard, . . . ), and the relative locations of the one or morehazardous conditions in the area, based at least in part on the sensordata and/or other data, as more fully described herein.

Based at least in part on the results of the analysis of the sensor dataand/or the other data, the IMC 108 can generate a layout of the area,including the objects and other features of the area, and the relativeand respective locations of the objects and other features in the areato each other, and/or including the one or more hazardous conditions (ifany) and their respective locations in the area, and/or can generate atravel route or travel guide (comprising the layout of the area) thatcan provide travel-related guidance (e.g., directions, such as GPS-typedirections) to the user that can enable the user to safely enter,navigate through, and/or exit the area, which may contain one or morehazardous conditions, in addition to the hazards or potential hazards auser may face from the objects or other features of the area as the usernavigates through the area with restricted or no visibility. The layout,travel route, and/or travel guide also can include a current location ofthe user relative to the respective locations of the objects, features,and/or hazardous conditions in the area.

The IMC 108 can facilitate presenting, via the display screen of theinterface component 106, the layout, travel route, and/or travel guideto the user, wherein the layout, travel route, and/or travel guide canbe or can appear to be overlaid on the user's field of view as the userviews the area (or portion of the area in the field of view) through thelens component of the ARD 102. Presenting the layout, travel route,and/or travel guide, the user's location, and the travel-relatedguidance (e.g., GPS-like directional or travel guidance) to the user canfacilitate enabling the user to know the user's location relative toobjects, barriers, and/or hazardous conditions in the area to enable theuser to safely enter, navigate through, or exit the area, which maycontain a hazard(s). This can be especially helpful if there isrestricted visibility in the area, for example, from smoke or loss oflight or power. This also can easily and advantageously be used byimpaired users (e.g., temporarily impaired users) who, for example, mayhave been impaired by an incident such as an incident that caused arcflash blindness or visual impairment to a user.

For example, as part of the layout, travel route, and/or travel guide,the IMC 108 can generate and facilitate presenting, via the displayscreen, visual information that can virtualize objects, barriers, and/orhazardous conditions in the area, and, even though the user may or maynot actually be able to see the actual objects, barriers, and/orhazardous conditions in the area, the user can perceive (e.g., see)virtualized versions of the objects, barriers, and/or hazardousconditions (and the respective and relative locations of such objects,barriers, and/or hazardous conditions) in the area relative to thelocation of the user to facilitate enabling the user to safely navigatearound the objects, barriers, and/or hazardous conditions in the area.In some implementations, the IMC 108 can generate and facilitatepresenting (e.g., emitting), via the audio component 304 and/or hapticcomponent 306, audible information (e.g., sounds, such as voicecommands, audible warnings or indicators) and/or or haptic information(e.g., ARD device vibrations or other type of haptic signals) tofacilitate guiding the user through the area and/or notifying or warningthe user when the user is in proximity to an object, barrier, and/orhazardous condition to enable the user to safely navigate around theobjects, barriers, and/or hazardous conditions in the area.

Referring briefly to FIG. 9 (along FIG. 1), FIG. 9 illustrates a diagramof example visual presentations 900 that can be presented to the user ofthe ARD 102 to enable the user to safely navigate through an area andaround objects, barrier, and/or hazardous conditions in the area, inaccordance with various aspects and embodiments of the disclosed subjectmatter.

The example visual presentations 900 can include visual presentation 902that can include customized information (e.g., augmented and/or virtualreality information) comprising an example travel guide that can includea layout of an area and travel guidance for the user to facilitateenabling the user to safely navigate through the area and aroundobjects, barrier, and/or hazardous conditions in the area, in accordancewith various aspects and embodiments of the disclosed subject matter.The travel guide, including the layout of the area, can be determined,and the visual presentation 902, comprising the travel guide, can begenerated by the IMC 108 based at least in part on the results ofanalyzing sensor data obtained from the sensor component 104 and/orother data, as more fully described herein.

As depicted in the travel guide of the visual presentation 902, the area904 (e.g., area of a building) can comprise rooms, such as rooms 906,908, 910, and 912, a hallway 914, a first door 916 at one end of thehallway 914, and a second door 918 at another end of the hallway 914.The respective rooms 906, 908, 910, and 912 can comprise respectiveobjects (e.g., cabinets, desks, furniture, terminals, devices, or systemcomponents, . . . ), wherein the respective objects comprise objects920, 922, 924, 926, 928, 930, 932, 934, and 936. The hallway 914 cancomprise objects 938, 940, 942, 944, and 946. The area 904 also caninclude a first hazard 948, such as a fire that has reduced visibilitysignificantly in the area 904, particularly in the hallway 914 and room906, where the user and associated ARD 102 are initially locate, whereinthe reduced visibility can make it difficult for the user to determine asafe way to exit the area 904, perceive the objects and hazards in thearea 904. The area 904 also can include a second hazard 950 (HZD), whichcan be a hazard due to a high voltage condition that can be hazardous tothe user.

The IMC 108 can determine the layout of the area 904, determine thelocation and/or orientation of the user and associated ARD 102, identifythe respective objects (e.g., 920 through 946) in the respective rooms(e.g., 906, 908, 910, and 912) and hallway 914 of the area 904 and therespective locations of the objects, identify the respective locationsof doors 916 and 918, and detect or determine the respective hazards 948and 950 in the hallway 914 of the area 904, based at least in part onthe results of analyzing the sensor data and/or the other data, as morefully described herein. Further, based at least in part on the resultsof analyzing the sensor data and/or the other data, includingdetermining the layout of the area 904, the objects, the hazards, etc.,the IMC 108 can determine and generate a travel guide that can comprisea travel route 952 that the user can travel to safely navigate throughthe area 904 and around any objects (e.g., 920 through 946), hazards(e.g., 948, 950), and/or other barriers (e.g., walls of the rooms orhallway) that the user may encounter as the user tries to exit the area904. The travel guide also can include certain information regarding thehazards 948 and 950. For example, the IMC 108 can include a hightemperature (T) indicator 954 (and/or fire indicator) with regard tohazard 948 (e.g., fire) to indicate that there is a high and/ordangerous temperature (and fire) in the region where that hazard 948 islocated; and the IMC 108 can include a high voltage (HV) indicator 956with regard to hazard 950 (e.g., high voltage of a component) toindicate that there is a high and/or dangerous voltage level in theregion where that hazard 950 is located. The IMC 108 can facilitatepresenting, via the display screen of the interface component 106, thetravel guide, comprising the layout of the area 904 and the travel route952, to the user.

As the user travels along, or at least attempts to travel along, thetravel route 952, the IMC 108 can determine and update the user'slocation and/or orientation to indicate a current location, orientation,and/or movement of the user in the area 904, based at least in part onsensor data from the sensor component 104 and/or other data, as morefully disclosed herein. If, for some reason, the user travels off of thetravel route 952, or conditions (e.g., hazards) in the area 904 change,the IMC 108 can determine (e.g., update) and facilitate presenting, viathe display screen, an updated travel route for the user to enable theuser to safely navigate through and exit the area 904.

In certain embodiments, based at least in part on the results ofanalyzing the sensor data and/or the other data, the IMC 108 also candetermine, generate, and facilitate presenting, via the display screenof the interface component 106, other customized information (e.g.,other augmented and/or virtual reality information) that can overlay, inthe field of view (e.g., through the lens component) of the user of theARD 102, other visual information relating to the area 904, the rooms(e.g., 906, 908, 910, and 912), the hallway 914, the doors (e.g., 916,918), the objects (e.g., 920 through 946), the hazards (e.g., 948, 950),and/or other barriers (e.g., walls of the rooms or hallway), as well asthe travel route 952.

As an example, the visual presentations 900 can comprise visualpresentation 960 that can be a field of view of the user as the userviews a portion of the area 904 (e.g., a portion of the hallway 914)through the lens component of the ARD 102 as the user travels andnavigates along the travel route 952. At this point, the user hastraveled along the travel route 952 from room 906 to the hallway 914 inbetween rooms 906 and 908, as depicted at reference numeral 958 in thevisual presentation 902. As visibility of the user has beensignificantly diminished by smoke from the hazard 948, the IMC 108 canenhance the view of the user through the ARD 102 by generating andfacilitating presenting virtual objects corresponding to the objects inthe hallway (e.g., hallway 914 in visual presentation 902) of the area904, visual indicators relating to hazards (e.g., hazard 950 in visualpresentation 902), and the travel route (e.g., travel route 952 invisual presentation 902) to enable the user to navigate around theobjects and hazards and exit through the door (e.g., door 918 in visualpresentation 902).

The field of view of the user through the ARD 102 in the visualpresentation 960 can comprise a portion of the hallway 914′ (a differentand/or virtual view of hallway 914), wherein virtual objects 942′, 944′,and 946′ (which can be a virtual view of objects 942, 944, and 946) canbe located in respective locations in the hallway 914′ in the field ofview perceived by the user via the display screen and lens component(e.g., over laid on the lens component) of the ARD 102. The field ofview in the visual presentation 960 also can include a virtual door 918′(which can be a virtual view of the door 918). The field of view in thevisual presentation 960 further can comprise a virtual hazard object950′ (corresponding to hazard 950) and high voltage indicator 956′(corresponding to high voltage indicator 956). The field of view in thevisual presentation 960 also can comprise a portion of the travel route952′ (corresponding to a portion of the travel route 952). From thefield of view in visual presentation 960 (presented to the user on theARD 102), the user can perceive and identify the virtual objects (e.g.,942′, 944′, and 946′), virtual hazard object 950′, travel route 952′,high voltage indicator 956′ associated with the virtual hazard object950′, and the virtual door 918′ (corresponding to the actual doorthrough which the user can exit from the hallway and the area 904). Theuser can use this information (e.g., virtual objects, virtual hazardobject, travel route, . . . ) presented in the visual presentation 960to safely navigate around the actual objects and actual hazards (andactual barriers, such as walls) in the hallway of the area 904 to exitthe door (e.g., 918), even if the visibility of the user is impaired,making it difficult or impossible for the user to perceive the objects,hazards, and barriers in the area 904.

In some embodiments, the ARD 102 can enable “over the shoulder” or“through the lens” viewing by a second or remote user (e.g., remoteexpert) to help guide or monitor the user for safety of the user of theARD 102 (including during a “man down” situation). There can beinstances where the user of the ARD is incapacitated, injured, and/orcan desire guidance from another person. To facilitate assisting orguiding the user of the ARD 102, a second or remote user, using anotherARD or another device, can communicate with the ARD 102 and/or the userof the ARD 102, and can access the IMC 108, sensor component 104,interface component 106, and/or other components of the ARD 102.

With access to the components of the ARD 102, the second or remote usercan use the other ARD or device can see the field of view of the user onthe other ARD or other device, have access to and see data available tothe user of the ARD 102 and/or presented on the display screen of theARD 102, manipulate controls (e.g., virtual controls) presented on theARD 102, and/or communicate commands to the ARD 102 for execution by theARD 102. This can enable the second or remote user to assess thesituation (e.g., visibility conditions, hazardous conditions, injury tothe user, . . . ) that the user of the ARD 102 is facing, and can enablethe second or remote user to facilitate guiding the user to safety fromany potential dangers associated with the situation. Such techniques andfeatures of the disclosed subject matter also can enable the second orremote user to review work tasks performed by the user to facilitateassessing how well the user has performed the work tasks and/or canenable the second or remote user to provide the user guidance orinstruction regarding how to perform a work task.

In certain implementations, the ARD 102 can detect and use the proximityof a user to a portion of a system (e.g., industrial automation system)to remove (e.g., discontinue or shutdown) power from and/or de-energizecircuits and systems upon approach of the user of the ARD 102 to theportion of the system. Based at least in part on the results ofanalyzing sensor data obtained from the sensor component and/or otherdata from another data source(s), the IMC 108 can determine thelocation, orientation, and/or motion of the user of the ARD 102,determine a layout of an area (e.g., an area where a system and/orfacility is located), including identifying objects, barriers, and/orhazardous conditions in the area and their respective and relativelocations to each other, and generate a map (e.g., 3-D map) of thelayout, as more fully described herein. The map can be based at least inpart on the design (e.g., the layout of machines, devices, processes,assets, . . . ) of the system or facility in the area. The ARD 102 canemploy the cameras (e.g., cameras with object recognition) of the imagecapture component 202, recognition component 212, environment component214, and/or other sensors to facilitate identifying the objects,barriers, and/or hazards in the area, and generating the map. In someimplementations, the cameras can be enhanced by having a 360 degree orfull spherical view of the area to facilitate enabling the user toperceive and avoid hazards (e.g., live electrical wires, a sharp object,an object which may cause the user to trip and fall, a ditch, aprecipice, . . . ) in any direction.

As the user is moving throughout the area, the IMC 108 of the ARD 102can continue to monitor and determine the location, orientation, and/ormotion of the user, based at least in part on sensor data received fromthe sensor component 104 and/or the other data, wherein sensors, such asnear field wireless sensors and/or other sensors (e.g., locationsensors, GPS system or sensors), of or associated with the sensorcomponent 104 can monitor and sense the location of the ARD 102 andassociated user, as more fully described herein. In some embodiments, inresponse to detecting the user in proximity to the portion of a system(e.g., industrial automation system), the IMC 108 can communicate amessage to another device to inform the other device that the user is inproximity to the portion of the system. In response to receiving themessage, the other device can remove (e.g., discontinue or shutdown)power from and/or de-energize the circuits and systems. In otherembodiments, in response to detecting the user in proximity to theportion of a system, the IMC 108 can remove power from and/orde-energize the circuits and systems.

The ARD 102 also can be employed to address issues relating to reviewingincidents. The ARD 102 can employ the image capture component 202 (e.g.,outward-facing camera(s)) and audio sensor component 204 to capture(e.g., record) video and audio of procedures or tasks (e.g., associatedwith a system) being performed. In some instances, during theperformance of the procedures or tasks, an incident (e.g., unacceptableprocedure or task performance, improved or optimal procedure or taskperformance, mistake, accident, injury or death, property damage,disaster) may occur.

Information (e.g., visual or audio information) in the recordedprocedure or task can be used by the ARD 102, another device, and/or aperson (e.g., a supervisor or expert) to review the procedure or task,and/or end product associated with (e.g., being produced by) a system.In some implementations, the ARD 102 can provide (e.g., communicate) thecaptured video and audio to devices of certain users (e.g., experts) whocan review the captured video and audio live as it is captured andcommunicated by the ARD 102 to the devices of the certain users or whocan review the captured video and audio at a later time. When anincident has occurred, the ARD user and/or certain users can review thevideo or audio to determine the correctness or appropriateness ofactions taken by people during the incident.

This can be helpful to facilitate avoiding such incident in the futureor better managing the response to such incident in the future. Also,information in the recorded procedure or task can be used to developimprovement suggestions to the process (e.g., like learning from a gamefilm). Information in the recorded procedure also can be used to reviewincidents for legal reasons (e.g., mistakes, accidents, or disasters).

In accordance with various embodiments, the ARD 102 can be used toenable the user of the ARD 102 to facilitate controlling machines from asafe distance from a machine or device associated with a system. The IMC108 of the ARD 102 or a user (e.g., system designer) can determine andgenerate an HMI, which can be a virtual HMI that can correspond to,and/or can be arranged to appear as, an actual HMI of a system, forexample, with regard to controls, data presented, and/or functionality,based at least in part on the results of the analysis of the sensor dataand/or other data (e.g., data from another data source(s), such as anexternal database). The IMC 108 can generate and facilitate presenting,via the display screen of the interface component 106, the virtual HMIto the user. Using the ARD 102, the user can utilize the virtual HMIfrom a desired distance away from the actual HMI it represents and/orfrom other components, which can include potentially hazardouscomponents (e.g., in proximity to the actual HMI). The user can viewdata presented on the virtual HMI displayed on the display screen of theARD 102 and/or can adjust virtual controls on the virtual HMI usinggestures, verbal commands, and/or textual or control commands, asdisclosed herein, to facilitate performing tasks associated with theactual HMI and associated system from a remote location that, forexample, can be a safe distance away from any potential hazardassociated with the actual HMI or system.

In some implementations, a virtual HMI or another virtual operatorinterface, which can be created in a manner that can be the same as orsimilar to the creation of the virtual HMI, as disclosed herein, can beemployed to facilitate performing tasks or operations in a manner thatcan be beneficial for hygiene and/or medical related purposes. Thevirtual operator interface can correspond to an actual operatorinterface and can be used to access data and perform tasks in a mannersame as or similar to how the actual operator interface accesses dataand performs tasks, wherein the virtual operator interface can beassociated with the actual operator interface. A user of the ARD 102 canuse the virtual operator interface (e.g., use or manipulate virtualbuttons, controls, levels, and/or parameters, etc., on the virtualoperator interface) to control the actual operator interface (e.g.,buttons, controls, levels, parameters, etc., on the actual operatorinterface) to perform tasks or operations without physically touchingthe actual operator interface (e.g., without touching buttons, controls,levels, parameters, etc., on the actual operator interface). This can bebeneficial for hygiene and/or medical related purposes (e.g., foodprocessing, medical equipment or procedures (e.g., medical equipmentand/or actual medical procedures, surgeries, intravenous medicineadministration, . . . )).

FIG. 10 illustrates a block diagram of an example system 1000 that canemploy an ARD in a communication network to facilitate performanceand/or enhancement of various activities and tasks, in accordance withvarious embodiments and aspects of the disclosed subject matter. Thesystem 1000 can comprise an ARD 1002. While the ARD 1002 is referencedas an augmented reality device, it is to be understood and appreciatedthat the ARD 1002 also can employ virtual reality functions andfeatures, in addition to augmented reality functions and features, asmore fully described herein.

The ARD 1002 can comprise a sensor component 1004, an interfacecomponent 1006, an IMC 1008, a bus component 1010, a processor component1012, and a data store 1014. The ARD 1002, the sensor component 1004,the interface component 1006, the IMC 1008, the bus component 1010, theprocessor component 1012, and the data store 1014 can respectively bethe same as or similar to, and/or can comprise the same or similarfunctionality as, respective components (e.g., respectively namedcomponents), as more fully described herein.

The ARD 1002 also can include a lens component 1016 that can comprise adesired number of lenses (e.g., one lens, two lenses, four lenses, orother desired number of lenses), wherein the lenses can be see-throughlenses that a user, who is interacting with or wearing the ARD 1002, cansee through the lenses to perceive objects that can be viewable by theuser using the user's eyes. In some implementations, one or more displayscreens of the display component (e.g., 302) can be associated with(e.g., integrated with) the lenses of the lens component 1016 tofacilitate presentation (e.g., display) of data (e.g., augmented realityinformation, images, and/or textual data, . . . ) to the user of the ARD1002.

The ARD 1002 further can comprise a communicator component 1018 (comm.component) that can facilitate communication of information between theARD 1002 and other devices via a wireless or wireline communicationconnection or channel. The communicator component 1018 can utilize oneor more communication technologies, such as a cellular technology, anInternet Protocol (IP)-based technology, wireless fidelity (Wi-Fi)technology, Wi-Max technology, gigabit wireless (Gi-Fi) technology,Hi-Fi technology (e.g., providing higher gigabit data communication thanGi-Fi or Wi-Fi), Bluetooth technology, ZigBee technology, near fieldcommunication (NFC) technology, near field communication interfaceprotocol (NFI), and/or other type of communication technology(ies) tofacilitate communicating data between the ARD 1002 and other devices.

The system 1000 can include a communication network 1020 that canfacilitate data communication between the ARD 1002 and other devicesassociated with the communication network 1020. In some implementations,the communicator component 1018 can facilitate communication of databetween the ARD 1002 and other devices in a communication networkenvironment via a communication connection with the communicationnetwork 1020. The communication network 1020 can comprise a macrocommunication network and/or a micro communication network. The macrocommunication network can be, can comprise, or can be associated with acore network, a cellular network, an IP-based network, Wi-Fi, Wi-Max,Gi-Fi network, Hi-Fi network, Bluetooth, ZigBee, etc. The microcommunication network can be associated with the macro communicationnetwork, wherein the micro communication network typically can operatein a defined local area (e.g., in or in proximity to a building, ahouse, a room, or other defined area). The micro communication networkcan be, can comprise, or can be associated with Wi-Fi, Wi-Max, Gi-Fi,Hi-Fi, Bluetooth, ZigBee, etc., and/or can be associated with (e.g.,connected to) the macro communication network. The micro communicationnetwork can be or can comprise, for example a local area network (LAN)or wireless LAN (WLAN), that can facilitate connecting certain devices(e.g., ARD 1002 and/or other devices) associated with the microcommunication network to each other and/or to the macro communicationnetwork. In accordance with various implementations, The macrocommunication network and/or a micro communication network can employradio communication, microwave communication, satellite communication,optical communication, sonic communication, electromagnetic inductioncommunication, or any other desired (e.g., suitable) communicationtechnology.

In some implementations, respective communication devices (e.g., ARD1002, data source device 1022, device 1024, and/or device 1026 (e.g.,communication device), . . . ) can be associated with (e.g.,communicatively connected to) the communication network 1020 via awireless communication connection (e.g., via a cell and associated basestation) or a wireline (e.g., wired) communication connection. Therespective communication devices (e.g., ARD 1002, data source device1022, device 1024, and/or device 1026, . . . ) can operate andcommunicate in the communication network environment. At various times,a communication device (e.g., ARD 1002, data source device 1022, device1024, and/or device 1026, . . . ) can be communicatively connected via awireless communication connection(s) to one or more radio accessnetworks (RANs) (not shown), which can comprise one or more basestations (not shown) to communicatively connect the communication deviceto the communication network 1020 to enable the communication device tocommunicate with other communication devices associated with (e.g.,communicatively connected to) the communication network 1020 in thecommunication network environment. The RANs can comprise, for example, a3GPP universal mobile telecommunication system (UMTS) terrestrial RAN(UTRAN), an E-UTRAN (e.g., Long Term Evolution (LTE) RAN), a GSM RAN(GRAN), and/or other type of RAN(s) employing another type ofcommunication technology.

The communication network 1020 can comprise one or more wirelinecommunication networks and one or more wireless communication networks,wherein the one or more wireless communication networks can be based atleast in part on one or more various types of communication technologyor protocols, such as, for example, 3G, 4G, 5G, or x generation (xG)network, wherein x can be virtually any desired integer or real value;Wi-Fi; Gi-Fi; Hi-Fi; etc. The communication network 18 can facilitaterouting voice and data communications between a communication device(s)(e.g., ARD 1002, data source device 1022, device 1024, and/or device1026, . . . ) and another communication device (e.g., another of the ARD1002, data source device 1022, device 1024, and/or device 1026, . . . )associated with the communication network 1020 in the communicationnetwork environment. The communication network 1020 also can allocateresources to the communication devices in the communication network1020, convert or enforce protocols, establish and enforce quality ofservice (QOS) for the communication devices, provide applications orservices in the communication network 1020, translate signals, and/orperform other desired functions to facilitate system interoperabilityand communication in the communication network 1020 (e.g., wirelessportion of the communication network 1020 or wireline portion of thecommunication network 1020). The communication network 1020 further cancomprise desired components, such as routers, nodes (e.g., generalpacket radio service (GPRS) nodes, such as serving GPRS support node(SGSN), gateway GPRS support node (GGSN)), switches, interfaces,controllers, etc., that can facilitate communication of data betweencommunication devices in the communication network environment.

As a communication device(s) (e.g., ARD 1002, data source device 1022,device 1024, and/or device 1026, . . . ) is moved through a wirelesscommunication network environment, at various times, the communicationdevice(s) can be connected (e.g., wirelessly connected) to one of aplurality of access points (APs) (e.g., macro or cellular AP, femto AP,pico AP, Wi-Fi AP, Gi-Fi AP, Hi-Fi AP, Wi-Max AP, hotspot (e.g., hotspot1.x, hotspot 2.x, wherein x is an integer number; communication device(e.g., communication device functioning as a mobile hotspot)) that canoperate in the wireless communication network environment. An AP (e.g.,a macro base station or micro base station) can serve a specifiedcoverage area to facilitate communication by the communication device(s)(e.g., ARD 1002, data source device 1022, device 1024, and/or device1026, . . . ) or other communication devices in the wirelesscommunication network environment. An AP can serve a respective coveragecell (e.g., macrocell, femtocell, picocell, . . . ) that can cover arespective specified area, and the AP can service mobile wirelessdevices, such as the communication device(s) (e.g., ARD 1002, datasource device 1022, device 1024, and/or device 1026, . . . ) located inthe respective area covered by the respective cell, where such coveragecan be achieved via a wireless link (e.g., uplink (UL), downlink (DL)).When an attachment attempt is successful, the communication device(s)(e.g., ARD 1002, data source device 1022, device 1024, and/or device1026, . . . ) can be served by the AP and incoming voice and datatraffic can be paged and routed to the communication device(s) throughthe AP, and outgoing voice and data traffic from the communicationdevice(s) can be paged and routed through the AP to other communicationdevices in the communication network environment. In some aspects, thecommunication device(s) (e.g., ARD 1002, data source device 1022, device1024, and/or device 1026, . . . ) can be connected and can communicatewirelessly using virtually any desired wireless technology, including,for example, cellular, Wi-Fi, Gi-Fi, Hi-Fi, Wi-Max, Bluetooth, WLAN,etc.

In some embodiments, the ARD 1002 can establish a direct communicationconnection (e.g., a direct wireline or wireless communicationconnection) with another device(s) (e.g., another ARD, a communicationdevice, . . . ) and can communicate with the other device(s) withoutusing the communication network 1020.

The system 1000 also can include one or more data source devices, suchas, for example, the data source device 1022, that can be associatedwith one or more data sources. The one or more data source devices(e.g., 1022) can be connected to the communication network 1020 tofacilitate establishing a communication connection with the ARD 1002and/or the device 1024. The one or more data sources, via the one ormore data source devices (e.g., 1022), can provide data (e.g., extrinsicdata) to the ARD 1002 and/or the device 1024 via the communicationnetwork 1020. The one or more data sources can comprise data relatingto, for example, industrial assets (e.g., industrial devices, industrialprocesses, . . . ), industrial automation systems, specification ortechnical information associated with industrial assets or otherdevices, products, product assembly, safety issues, security issues,training (e.g., job or task training), design assistance (e.g.,industrial or HMI design or configuration assistance), weatherconditions, environmental conditions, hazardous conditions, emergencyconditions, language translation, literature, music, and/or news, etc.

The ARD 1002 (e.g., the IMC 1008 of the ARD 1002) and/or the device 1024(e.g., an IMC 1028 of the device 1024) can utilize and analyze the data(e.g., extrinsic data) from the one or more data sources, along withother data sensed or otherwise obtained by the ARD 1002 or the device1024, to facilitate determining or generating desired data (e.g.,customized data, such as augmented reality data) that can be presentedto the user via the ARD 1002 (e.g., via the interface component 1006 ofthe ARD 1002).

In some implementations, the device 1024 can comprise the IMC 1028. TheIMC 1028 can comprise all or a desired portion of the components,functions, and/or features of the IMC 1008 of the ARD 1002, and/or canhave the capability to perform all or a desired portion of theoperations and/or tasks that can be performed by the IMC 1008 of the ARD1002. In some embodiments, the IMC 1008 of the ARD 1002 and the IMC 1028of the device 1024 can communicate with each other to coordinateperformance of operations or tasks.

For example, the IMC 1008 of the ARD 1002 can communicate and coordinatewith the IMC 1028 of the device 1024 to have the IMC 1028 performcertain operations or tasks on behalf of the IMC 1008. This can be doneto, for example, offload the certain operations or tasks to the IMC 1028to (e.g., free up resources of the IMC 1008 to) enable the IMC 1008 toutilize some of its resources to perform other operations or tasks,and/or to enable parallel performance of operations or tasks (e.g.,parallel processing of data) by the IMC 1008 and the IMC 1028, which canincrease the speed at which desired data (e.g., customized data, such asaugmented reality data) can be presented to the user via the ARD 1002.The IMC 1028 (or another component) of the device 1024 can communicatedata relating to the performance of the certain operations or tasks tothe IMC 1008 of the ARD 1002, via the communication network 1020, forfurther processing by the IMC 1008 (if further processing is desired ornecessary) or for presentation (e.g., via the interface component 1006of the ARD 1002) to the user of the ARD 1002.

In some implementations, the ARD 1002 can be employed by the user tohave the ARD 1002 communicate with, interact with, and/or control (e.g.,communicate with, remotely interact with and/or control) one or moredevices, such as, for example, the device 1026 via the communicationnetwork 1020 and/or via a direct communication connection (e.g., directwireless or wireline communication connection) between the ARD 1002 andthe one or more devices (e.g., 1026). The one or more devices (e.g.,1026) can be located in an area 1030 where the user is located or thatis in proximity to the location of the user and associated ARD 1002, orcan be located in another area 1032 that is remote from the location ofthe user and associated ARD 1002. The one or more devices (e.g., 1026)can be, can be part of, and/or can be associated with a robotic device,a drone, a SCADA system, an industrial device or other type of device,an industrial process, a network component that facilitatescommunication network communications, a house, a building, another typeof structure, law enforcement, a fire department, an emergency responseentity, a hazard response entity, and/or another entity.

The ARD 1002, and the user via the ARD 1002, can communicate with,interact with, and/or control (e.g., communicate with, remotely interactwith and/or control) the device 1026 to monitor operations andconditions of the device 1026 and/or components associated with or inproximity to the device 1026, obtain data regarding operations andconditions of the device 1026 and/or components associated with or inproximity to the device 1026, modify one or more parameters of thedevice 1026, detect a hazardous condition of or associated with thedevice 1026 and/or components associated with or in proximity to thedevice 1026, adjust parameters or operation of the device 1026 tomitigate (e.g., alleviate, reduce, or eliminate) the hazardouscondition, utilize and control the device 1026 to perform operations ortasks (e.g., perform an assembly task to facilitate assembling aproduct, perform a task to mitigate a hazardous condition), and/or haveother interactions with or control over the device 1026, without theuser having to touch or physically view the device 1026 and/orcomponents associated with or in proximity to the device 1026.

For instance, from the location where the ARD 1002 and user are located,the ARD 1002 can be employed by the user to communicate with, interactwith, and control (e.g., remotely communicate with, interact with, andcontrol) the device 1026 to perform (e.g., remotely perform) tasks inthe area (e.g., area 1030 or area 1032) where the device 1026 islocated. For example, there can be an oil pump (e.g., a relativelydangerous oil pump) in an area (e.g., area 1032). It can be desirablefor the user of the ARD 1002 to control operations of the oil pump inthat area (e.g., area 1032) without the user having to physically go tothe oil pump and perform tasks to control operations of the oil pump, inorder to keep the user from being subjected to the potential dangers ofbeing in physical proximity to the oil pump.

The user can employ the ARD 1002 and the device 1026, which can be, forexample, a drone or other robotic device, to perform the tasks withrespect to the oil pump without the user being in physical proximity tothe oil pump. The user, through voice commands, written commands, and/orgestures representing commands to the ARD 1002, can use the ARD 1002 toperform operations and tasks to control (e.g., remotely control) thedevice 1026, to have the device 1026 move (e.g., navigate, fly, orotherwise travel) to the oil pump and perform the tasks on the oil pumpthat the user desires to have performed to enable suitable and safeoperation of the oil pump. For example, the device 1026 can be a droneand the user can employ the ARD 1002 to control the drone to have thedevice 1026 (e.g., drone) move (e.g., fly) to the oil pump and performthe tasks on the oil pump. As another example, the device 1026 can be arobotic device, and the ARD 1002 can control a drone to have the dronetransport the device 1026 (e.g., robotic device) to the oil pump, andcan control the device 1026 (e.g., robotic device) to perform the taskson the oil pump.

In connection with controlling the operation of the device 1026, the ARD1002 can receive information (e.g., images, sensor data from sensors ofthe device 1026, audio information, . . . ) from the device 1026 and/orinformation (e.g., images, sensor data from other sensors, audioinformation, . . . ) from one or more other data sources (e.g., via datasource device 1022 and/or the IMC 1028 of device 1024), wherein suchinformation can relate to the operation of the device 1026, theoperation or conditions of the oil pump, and/or the operation orconditions of other components associated with (e.g., in proximity to)the oil pump.

The IMC 1008 of the ARD 1002 (and/or the IMC 1028) can determine data(e.g., customized data, such as augmented reality information) that canuseful and relevant to the user in using the ARD 1002 to controloperations of the device 1026 to perform the desired tasks on the oilpump. For example, automatically, dynamically, or based at least in parton instructions or feedback from the user, the IMC 1008 (and/or the IMC1028) can determine that images (e.g., live images provided by thedevice 1026, with other relevant data, such as, a heat map of the oilpump, a hazard warning indicator (if any) associated with the oil pump,parameter information (e.g., in textual form or in graphic or meterform) associated with the oil pump, virtual objects associated with theoil pump (e.g., objects inside the oil pump that are not visible in theimages), and/or an abnormal data indicator (if any) associated with theoil pump, are to be presented on the display screen(s) of the ARD 1002for observation and use by the user, in accordance with the definedinformation management criteria and/or user preferences. The IMC 1008can facilitate presenting the customized data, comprising the augmentedreality data and/or virtual reality data (e.g., the images and the otherrelevant data), on the display screen(s) of the ARD 1002 to the user.The ARD 1002 can update (e.g., modify) the images and the other relevantdata associated with the oil pump presented on the display screen(s) ofthe ARD 1002 to the user in real time, or at least in substantially realtime (e.g., on a sub-second basis). The user can use such data presentedon the display screen(s) of the ARD 1002 to decide what action to takewith regard to performing tasks with respect to the oil pump. The usercan communicate voice, textual, and/or gesture commands, instructions,or requests to the interface component 1006 of the ARD 1002 to controloperation of the ARD 1002 to have the ARD 1002 control operations andthe performance of tasks by the device 1026 on or with respect to theoil pump.

FIG. 11 depicts a block diagram of an example IMC 1100, in accordancewith various embodiments and aspects of the disclosed subject matter. Insome embodiments, the IMC 1100 can part of an ARD associated with auser. In other embodiments, the IMC 1100 can part of a device that cancommunicate with an IMC of the ARD to facilitate presenting desired data(e.g., customized data, such as augmented reality data) to the user ofthe ARD.

The IMC 1100 can comprise, for example, a communicator component 1102,an operations manager component 1104, a recognition component 1106, adata determination component 1108, an emphasis component 1110, aprocessor component 1112, and a data store 1114.

The communicator component 1102 can communicate data between the IMC1100 and one or more other components or devices, such as, for example,other components (e.g., the sensor component, the interface component,the bus component, . . . ) of the ARD, and devices (e.g., communicationdevice, data source device, . . . ) associated with one or more entities(e.g., another user(s), a data source(s), an emergency response entity,law enforcement, a fire department, . . . ). The communicator component1102 can communicate information using wireline or wirelesscommunication technologies and protocols, as more fully describedherein. The communicator component 1102 also can facilitatecommunicating desired data (e.g., customized data, such as augmentedreality data) for presentation, via the ARD, to a user.

The operations manager component 1104 can control (e.g., manage)operations associated with the IMC 1100. For example, the operationsmanager component 1104 can facilitate generating instructions to havecomponents of the IMC 1100 perform operations, and can communicaterespective instructions to respective components (e.g., communicatorcomponent 1102, recognition component 1106, data determination component1108, emphasis component 1110, . . . ) of the IMC 1100 to facilitateperformance of operations by the respective components of the IMC 1100based at least in part on the instructions, in accordance with thedefined information management criteria and the defined informationmanagement algorithm(s). The operations manager component 1104 also canfacilitate controlling data flow between the respective components ofthe IMC 1100 and controlling data flow between the IMC 1100 and anothercomponent(s) (e.g., the sensor component, the interface component,and/or the bus component, . . . ) of the ARD or a device(s) (e.g.,another ARD; a device, such as a communication device, associated with aremote location; a data source device(s)) associated with (e.g.,connected to) the IMC 1100.

The recognition component 1106 can employ one or more recognitiontechniques to detect and/or determine characteristics, features, orconditions of entities or objects to facilitate recognizing oridentifying entities or objects. The recognition component 1106 cancomprise all or at least a portion of the capabilities and functionalityof the recognition component of the sensor component, as more fullydescribed herein, and/or can comprise other capabilities andfunctionality beyond that of the recognition component of the sensorcomponent. In some implementations, the recognition component of thesensor component can sense or detect characteristics, features, orconditions of entities or objects. The recognition component of thesensor component can recognize or identify the entities or objects basedat least in part on the characteristics, features, or conditions ofentities or objects. Additionally or alternatively, the recognitioncomponent 1106 can analyze the sensor data relating to thecharacteristics, features, or conditions of entities or objects sensedby the sensor component and/or at least partially analyzed by therecognition component of the sensor component, and the recognitioncomponent 1106 can recognize or identify the entities or objects basedat least in part on the sensor data (e.g., the detected characteristics,features, or conditions of entities or objects).

The data determination component 1108 can analyze data, including sensordata obtained from the sensor component, data obtained from other datasources, user preference data from a user profile or from instructionsfrom the user, and/or other data. Based at least in part on the resultsof that analysis, the data determination component 1108 can determine aset of customized data, such as augmented reality data and/or virtualreality data, to present via the interface component (e.g., displayscreen(s), audio component, haptic component, . . . ) of the ARD to theuser, in accordance with the defined information management criteria.The data determination component 1108 can determine what types of datato present (e.g., images (e.g., photographs, IR images, . . . ), virtualimages or objects, textual data, audio data, and/or haptic feedbackdata) via the interface component of the ARD to the user, whatrespective forms respective items of data are to be presented (e.g.,present heat or temperature related data in the form of a heat map;present warning or hazard indicators when abnormal data or hazards aredetected; present parameter data in text form or virtual meter form; . .. ) via the interface component of the ARD to the user, what respectivelocations, on a display screen(s) associated with the lens component ofthe ARD, the respective items of data are to be presented (e.g., todesirably correspond to the respective locations of objects, entities,environment, etc., that the user can perceive in the user's field ofview through the lens component of the ARD and/or on the displayscreen(s) of the ARD), etc.

The emphasis component 1110 can emphasize or highlight information(e.g., visual information) presented on a display screen(s) of the AR,information presented in audio form via the audio component, orinformation otherwise presented via the interface component of the AR tothe user. For example, the user can be performing assembly tasks inconnection with assembly of a product, the data determination component1108 can determine the next part to be installed as part of the productassembly, based at least in part on analysis of sensor data obtainedfrom the sensor component and/or other data obtained from other datasources. Based at least in part on the determination of the next part tobe installed, as the user views through the lens component of the AR,the emphasis component 1110 can emphasize (e.g., highlight) the nextpart, for example, in a part bin of various parts relative to the otherparts in the part bin to facilitate identification and selection of thenext part by the user.

For instance, on a display screen associated with the lens component ofthe AR, the emphasis component 1110 can generate and present a visualindicator (e.g., a colored circular or rectangular ring) on the displayscreen in proximity to or surrounding the location on the display screenwhere the user perceives (e.g., sees) the next part through the lenscomponent and associated display screen to emphasize or highlight thenext part on the display screen for correct selection by the user. Asanother example, additionally or alternatively, on the display screenassociated with the lens component of the AR, the emphasis component1110 can magnify the next part (and/or the location of the bincontaining the next part) to enlarge the next part (or associated bin)relative to the other parts (or other bins) and/or can otherwisede-emphasize other parts in the part bin (and/or de-emphasize other binsin the part bin) (e.g., by magnifying and/or maintaining clear view ofthe next part (and associated bin) and blurring or otherwisede-emphasizing the other parts (or other bins)). Conversely, based atleast in part on data analysis results from the data determinationcomponent 1108 indicating the part the user is attempting to select forinstallation on the product is not correct, the emphasis component 1110can emphasize when the user is attempting to select the wrong part forinstallation as the next part on the product by facilitating presentinga negative indicator (e.g., a red circle with a line through the circleon the display screen of the ARD) over the part (e.g., in acorresponding location on the display screen and lens component, asviewed by the user) to notify the user that the attempted selection ofthe part is incorrect.

The emphasis component 1110 also can highlight or otherwise emphasize alocation on a product (e.g., via emphasis of a corresponding location onthe display screen and lens component of the ARD) where the next part isto be installed on the product, based at least in part on adetermination of such location (and corresponding location) by the datadetermination component 1108 (e.g., based at least in part on results ofdata analysis of sensor data and/or other data by the data determinationcomponent 1108). Conversely, based at least in part on data analysisresults from the data determination component 1108 indicating an attemptto install the next part at a particular location on the product is notcorrect, the emphasis component 1110 can emphasize when the user isattempting to install the next part in the wrong location on the productby facilitating presenting a negative indicator (e.g., a red circle witha line through the circle on the display screen of the ARD) over thepart and/or attempted location (e.g., in a corresponding location(s) onthe display screen and lens component, as viewed by the user) to notifythe user that the attempted location on the product is the wronglocation to install the part.

The processor component 1112 can operate in conjunction with the othercomponents (e.g., communicator component 1102, operations managercomponent 1104, recognition component 1106, data determination component1108, emphasis component 1110, data store 1114, . . . ) to facilitateperforming the various functions and operations of the IMC 1100. Theprocessor component 1112 can employ one or more processors (e.g., CPUs,GPUs, FPGAs, etc.), microprocessors, or controllers that can processdata, such as data relating to a user of the ARD, data relating to anenvironment in which the user and the ARD are located,maintenance-related data, safety-related data, security-related data,design-related data (e.g., data relating to design of an industrialautomation system, or data relating to design of a product), HMIdesign-related data, asset-related data (e.g., device data, processdata, asset data, system data, etc.) associated with systems (e.g.,industrial automation systems) or products, customer or client relateddata, data relating to parameters associated with systems or products,algorithms (e.g., algorithm(s) relating to recognizing or identifyingrespective industrial devices, industrial processes, industrial assets,network-related devices, interrelationships between such devices,processes, or assets), etc., to facilitate performing operations andtasks (e.g., in connection with an industrial automation system(s), orin connection with another type of environment); and can control dataflow between the IMC 1100 and other components or devices associatedwith the IMC 1100.

In accordance with various aspects, the data store 1114 can store datastructures (e.g., user data, metadata); code structure(s) (e.g.,modules, objects, classes, procedures), commands, or instructions; datarelating to a user of the ARD, data relating to an environment in whichthe user and the ARD are located, maintenance-related data,safety-related data, security-related data, design-related data (e.g.,data relating to design of an industrial automation system, or datarelating to design of a product), HMI design-related data, asset-relateddata (e.g., device data, process data, asset data, system data, etc.)associated with systems (e.g., industrial automation systems) orproducts, customer or client related data, data relating to parametersassociated with systems or products, algorithms (e.g., algorithm(s)relating to recognizing or identifying respective industrial devices,industrial processes, industrial assets, network-related devices,interrelationships between such devices, processes, or assets), and soon. The processor component 1112 can be functionally coupled (e.g.,through a memory bus of a bus component) to the data store 1114 in orderto store and retrieve data desired to operate and/or conferfunctionality, at least in part, to the communicator component 1102,operations manager component 1104, recognition component 1106, datadetermination component 1108, emphasis component 1110, etc., of the IMC1100 and/or substantially any other operational aspects of the IMC 1100.

The aforementioned systems and/or devices have been described withrespect to interaction between several components. It should beappreciated that such systems and components can include thosecomponents or sub-components specified therein, some of the specifiedcomponents or sub-components, and/or additional components.Sub-components could also be implemented as components communicativelycoupled to other components rather than included within parentcomponents. Further yet, one or more components and/or sub-componentsmay be combined into a single component providing aggregatefunctionality. The components may also interact with one or more othercomponents not specifically described herein for the sake of brevity,but known by those of skill in the art.

Embodiments, systems, and components described herein, as well asindustrial automation or control systems and industrial automationenvironments in which various aspects set forth in the subjectspecification can be carried out, can include computer or networkcomponents such as servers, clients, programmable logic controllers(PLCs), automation controllers, communications modules, mobilecomputers, wireless components, control components and so forth whichare capable of interacting across a network. Computers and serversinclude one or more processors—electronic integrated circuits thatperform logic operations employing electric signals—configured toexecute instructions stored in media such as random access memory (RAM),read only memory (ROM), a hard drive(s), as well as removable memorydevices, which can include memory sticks, memory cards, flash drives,external hard drives, and so on.

Similarly, the term PLC or automation controller as used herein caninclude functionality that can be shared across multiple components,systems, and/or networks. As an example, one or more PLCs or automationcontrollers can communicate and cooperate with various network devicesacross the network. This can include substantially any type of control,communications module, computer, Input/Output (I/O) device, sensor,actuator, and human machine interface (HMI) that communicate via thenetwork, which includes control, automation, and/or public networks. ThePLC or automation controller can also communicate to and control variousother devices such as I/O modules including analog, digital,programmed/intelligent I/O modules, other programmable controllers,communications modules, sensors, actuators, output devices, and thelike.

The network can include public networks such as the internet, intranets,and automation networks such as control and information protocol (CIP)networks including DeviceNet, ControlNet, and Ethernet/IP. Othernetworks include Ethernet, DH/DH+, Remote I/O, Fieldbus, Modbus,Profibus, CAN, wireless networks, serial protocols, and so forth. Inaddition, the network devices can include various possibilities(hardware and/or software components). These include components such asswitches with virtual local area network (VLAN) capability, LANs, WANs,proxies, gateways, routers, firewalls, virtual private network (VPN)devices, servers, clients, computers, configuration tools, monitoringtools, and/or other devices.

In view of the example systems and/or devices described herein, examplemethods that can be implemented in accordance with the disclosed subjectmatter can be further appreciated with reference to flowcharts in FIGS.12-15. For purposes of simplicity of explanation, example methodsdisclosed herein are presented and described as a series of acts;however, it is to be understood and appreciated that the disclosedsubject matter is not limited by the order of acts, as some acts mayoccur in different orders and/or concurrently with other acts from thatshown and described herein. For example, a method disclosed herein couldalternatively be represented as a series of interrelated states orevents, such as in a state diagram. Moreover, interaction diagram(s) mayrepresent methods in accordance with the disclosed subject matter whendisparate entities enact disparate portions of the methods. Furthermore,not all illustrated acts may be required to implement a method inaccordance with the subject specification. It should be furtherappreciated that the methods disclosed throughout the subjectspecification are capable of being stored on an article of manufactureto facilitate transporting and transferring such methods to computersfor execution by a processor or for storage in a memory.

FIG. 12 illustrates a flow chart of an example method 1200 that candetermine and present customized information, comprising augmentedreality information, to a user to facilitate performance of tasks by theuser, in accordance with various aspects and embodiments of thedisclosed subject matter. The method 1200 can be employed by, forexample, an IMC of an ARD.

At 1202, sensor data relating to an environment and/or other data fromanother data source(s) can be received. The IMC can receive the sensordata from one or more sensors of the sensor component and/or can receiveother data from one or more other data sources. The one or more otherdata sources can comprise the data store of the ARD and/or one or moreexternal data sources that are external to the ARD, but can be accessedby the ARD (e.g., via a communication network or a direct communicationconnection). The sensor data can relate to, for example, the environmentassociated with (e.g., surrounding) the ARD. The other data obtainedfrom the one or more other data sources can relate directly to theenvironment and/or can otherwise be relevant to the environment or thetasks to be performed or activities of the user of the ARD.

At 1204, the sensor data and/or the other data can be analyzed. The IMCcan analyze the sensor data and/or the other data to generate analysisresults, to facilitate determining customized information (e.g.,augmented reality information and/or virtual reality information) topresent, via the ARD, to the user.

At 1206, customized information, comprising augmented realityinformation and/or virtual reality information, can be determined basedat least in part on the results of the analysis. The IMC can determinethe customized information, comprising the augmented reality informationand/or the virtual reality information, based at least in part on theresults of the analysis of the sensor data and/or other data, inaccordance with the defined information management criteria and/or userpreferences or instructions of the user.

At 1208, the customized information, comprising the augmented realityinformation and/or the virtual reality information, can be presented,via the ARD, to the user. The IMC can control the interface component tofacilitate presenting the customized information, comprising theaugmented reality information and/or the virtual reality information,via the interface component of the ARD, to the user. For example, asdetermined by the IMC, the customized information can be presented via adisplay screen of the display component of the interface component,presented via the audio component (e.g., speakers) of the interfacecomponent, and/or presented via the haptic component of the interfacecomponent, to the user.

FIG. 13 presents a flow chart of an example method 1300 that candetermine and present customized information, comprising augmentedreality information, to facilitate generating a map comprising a travelroute that includes indicators for hazards and/or obstacles, inaccordance with various aspects and embodiments of the disclosed subjectmatter. The method 1300 can be employed by, for example, an IMC of anARD and/or another component(s) of the ARD.

At 1302, destination information, which can indicate a destination towhich a user desires to travel, can be received. The user can submit thedestination information via an interface component, wherein thedestination information can specify a destination to which a userdesires to travel. The user can provide the destination information tothe interface component using a keyboard of the interface component, byselecting a link relating to the destination, by speaking thedestination information into a microphone of the interface component, orin another desired way. The IMC can receive the destination informationfrom the user via the interface component.

The destination can be a street location (e.g., a house or a buildinglocated on a street), a place (e.g., a particular location in a park; aparticular location in or on a desert, a mountain, or a wooded area; aparticular location on a body of water), a location inside a house orbuilding. The destination information can comprise an address (e.g.,street address), a name of the destination, a room number, a room name,a floor number, and/or other information that can indicate thedestination.

At 1304, a destination location can be determined based at least in parton the destination information. The IMC can determine or facilitatedetermining (e.g., a navigation or GPS system of the sensor component)the destination location based at least in part on the destinationinformation.

At 1306, a starting location of the user can be determined. The startinglocation of the user can be the current location of the user or anotherlocation that can be indicated by the user. The IMC or a sensorcomponent (e.g., the navigation or GPS system) can determine the currentlocation of the user based at least in part sensor information relatingto location of the user (e.g., location of the ARD of the user) that issensed by the sensor component. As desired, the user can provideorigination information for another location, different from the currentlocation, to use as a starting location via the interface component in amanner similar to providing the destination information. The IMC candetermine the starting location based at least in part on theorigination information received from the user.

At 1308, one or more hazards or obstacles in an area associated with thestarting location and the destination location can be determined. TheIMC can determine the one or more hazards or obstacles based at least inpart on sensor information received from one or more sensors of thesensor component.

At 1310, a travel route can be determined based at least in part on thestarting location of the user, the destination location, and the one ormore hazards or obstacles. The IMC can determine or facilitatedetermining (e.g., via the navigation or GPS system) the travel routebetween the starting location and the destination location based atleast in part on the respective locations of the starting location andthe destination location, and the one or more hazards or obstacles. TheIMC can tailor or facilitate tailoring the travel route based at leastin part on whether the user will be walking, riding a non-motorizedvehicle (e.g., bicycle), riding in a motor vehicle (e.g., automobile),and/or whether the user has a disability and, if so, the type ofdisability (e.g., blind, deaf). The IMC also can tailor or facilitatetailoring the travel route to facilitate avoiding or at leastsubstantially avoiding the one or more hazards or obstacles.

At 1312, the travel route can be presented to the user via the interfacecomponent of the ARD. The IMC can facilitate presenting customizedinformation (e.g., augmented reality information and/or virtual realityinformation), comprising the travel route, to the user via the interfacecomponent of the ARD associated with the user. The travel route canprovide a map of the travel route, textual directions of the travelroute, and/or audio directions of the travel route. The travel routeinformation, such as the map, textual directions, and/or audiodirections, can provide information regarding the one or more hazards orobstacles. For example, the map can include one or more indicators atrespective locations on the map, wherein the one or more indicators canindicate the respective locations of the one or more hazards orobstacles, the type of hazard or obstacle, a level of danger of a hazardor obstacle, and/or other information regarding the one or more hazardsand obstacles.

As the user proceeds from the starting location to the destinationlocation, the IMC can update (e.g., modify or adjust) the travel routeinformation (e.g., update the customized information), for example, toindicate a current location of the user relative to the startinglocation and destination location, indicate whether the user iscurrently on the travel route or has deviated from the travel route,update (e.g., modify or adjust) the travel route based at least in parton the current location or the user relative to the destination locationand/or a deviation from the travel route, update the travel route basedat least in part on a newly discovered hazard or obstacle, and/or notifythe user of a new hazard or obstacle (e.g., a “Don't Walk” light at atraffic intersection along the travel path, a curb on a street the useris crossing, or a hostile person or criminal with a weapon, . . . ). TheIMC can facilitate presenting, via the interface component of the ARD,the updated travel route information (e.g., updated travel route andmap) to the user.

FIG. 14 depicts a flow chart of an example method 1400 that candetermine and present customized information, comprising augmentedreality information, to facilitate performance of product assemblytasks, in accordance with various aspects and embodiments of thedisclosed subject matter. The method 1400 can be employed by, forexample, an IMC of an ARD.

At 1402, a task relating to assembly of a product can be determined,based at least in part on assembly information relating to the productreceived from a data store, a sensor component, and/or a user. The IMCcan receive the assembly information, comprising task-relatedinformation, from the data store, the sensor component (e.g., one ormore sensors) and/or the user. The IMC can analyze the assemblyinformation. Based at least in part on the results of the analysis, theIMC can determine the task (e.g., next task) relating to assembly of theproduct. The assembly information can comprise directions and imagesrelating to assembling the product, including information regarding therespective parts of the product, the order of installation of parts ofthe product, respective parameters of respective parts, any notableconsiderations, conditions, or hazards associated with assembling theproduct, and/or other types of assembly information. Notableconsiderations, conditions, or hazards associated with assembling theproduct can comprise, for example, that part “A” is sharp and is to behandled with care, part “B” is fragile and is to be handled with care,installation of part “C” on the product involves high voltage andpotential for electrical shock, so suitable care and precautions shouldbe taken, part “D” has to be installed before part “E” is installed,and/or other considerations, conditions, or hazards associated withassembling the product.

At 1404, a part to be installed as part of the task relating to assemblyof the product can be determined. The IMC can identify or determine thepart to be installed as part of the assembly task based at least in parton the assembly information relating to assembly of the product, whichcan be stored in the data store.

At 1406, customized information (e.g., augmented reality information),comprising part information, which can indicate the part to be chosenfor installation as part of the task, can be presented to the user. TheIMC can generate and/or facilitate presenting, via the interfacecomponent of the ARD associated with the user, the customizedinformation, comprising the part information, to the user. Thecustomized information (e.g., augmented reality information), comprisingthe part information, can include audio information regarding the partinformation presented via the interface component (e.g., speaker(s) ofthe ARD), and/or can include a visual presentation regarding the partinformation (e.g., a drawing of the part (e.g., in space), and/or thepart being visually highlighted or emphasized by color, by flashing,and/or by other emphasis means), for example, as if the part is present(e.g., virtually present) in the live view of the assembly of theproduct (e.g., partial assembly comprising the part) that can beviewable through the ARD by the user. The customized information,comprising the part information, can facilitate instructing the user asto which part to select (e.g., choose) for installation at this point inthe assembly process.

At 1408, in response to selection of the part by the user, adetermination can be made regarding whether the user selected thecorrect part. In response to selection of the part by the user, the IMC,employing the sensor component, can determine whether the user selectedthe correct part. The sensor component (e.g., camera) of the ARD candetect the part selected (e.g., picked up and held) by the user and canprovide sensor information relating to detection of the part selected bythe user to the IMC. The IMC can determine the part that was selected bythe user based at least in part on the sensor information and partidentification information obtained from a data store, wherein the partidentification information can facilitate identifying the selected part.For example, the IMC can compare an image of the part captured by thesensor component to images of parts stored in the data store and candetermine an image of a part in the data store that matches or at leastsubstantially matches the image of the part captured by the sensorcomponent, wherein the image of the part in the data store can beassociated with identification information that can identify the partand can identify where and when the part is to be installed in theassembly process.

If, at 1408, it is determined that the user selected the correct part,at 1410, a confirmation indicator or message that the correct part wasselected can be presented. For instance, if the IMC determines that theuser selected the correct part, the IMC can generate and facilitatepresenting, via the interface component (e.g., a display screen and/oran audio speaker of the ARD), the confirmation indicator or message. Theconfirmation indicator or message, for example, can be a colored (e.g.,green) indicator that can highlight or emphasize the selected part(e.g., the selected part can be outlined in green in the visual displayof the ARD) to indicate the correct part was selected and/or voiceinformation confirming that the correct part was selected can becommunicated via the speaker to the user.

If, at 1408, it is determined that the user did not select the correctpart, at 1412, an incorrect part indicator or message can be presented(e.g., via the ARD). If the IMC determines that the user did not selectthe correct part, the IMC can generate and facilitate presenting, viathe interface component (e.g., the display screen and/or audio speakerof the ARD), the incorrect part indicator or message. The incorrect partmessage, for example, can be a colored (e.g., red) indicator that canhighlight or emphasize the selected part (e.g., the selected part can beoutlined in red in the visual display of the ARD) to indicate the wrongpart was selected by the user and/or voice information indicating thatthe wrong part was selected can be communicated via the speaker to theuser.

At 1414, in response to determining the part to be installed as part ofthe task, the location where the part is to be installed on the productcan be emphasized (e.g., highlighted) based at least in part on thelocation, view, and/or orientation of the ARD, assembly informationrelating to the product, identification or orientation of the part,and/or identification or orientation of one or more other parts that canbe in proximity to the part on the product. In response to determiningthe part to be installed as part of the task, the IMC can emphasize, viaa display screen of the ARD, the location on the product where the partis to be installed based at least in part on the location, view, and/ororientation of the ARD, the assembly information relating to theproduct, the identification or orientation of the part, and/or theidentification or orientation of the one or more other parts inproximity to the part on the product. The emphasis provided on thedisplay screen of the lens component of the ARD can be, for example, anemphasis or highlight, such as a colored (e.g., green) and shaped (e.g.,circular or rectangular shaped) indicator around (e.g., surrounding) thelocation, an arrow indicator pointing at the location, and/or a messagein proximity to the location.

The IMC, employing sensors of the sensor component, can determine thelocation, orientation, motion, and field of view of the ARD of the user.In some implementations, the IMC can retrieve assembly informationrelating to the product from the data store, and can determine (e.g.,automatically) the location on the product where the part is to beinstalled based at least in part on the assembly information. The IMCcan determine the position in the field of view on the display screen ofthe lens component of the ARD that corresponds to (e.g., aligns with)the location where the part is to be installed. The IMC can emphasizethat position in the display screen of the ARD to facilitate emphasizingthe corresponding location where the part is to be installed on theproduct, wherein, if the position changes due, for example, to the usermoving the ARD (e.g., the user moving his head and the ARD thereon), theIMC can move the position on the display screen to continue to have theposition on the display screen, and, accordingly, the emphasis on thedisplay screen of the ARD, correspond to the location where the part isto be installed on the product.

In other implementations, the IMC can determine (e.g., dynamicallyand/or automatically) the location where the part is to be installedbased at least in part on the results of an analysis of the part andanalysis of other parts associated with the product. For instance, theIMC, employing sensors of the sensor component, can employ objectrecognition to determine the part and the one or more other parts of theproduct that are in proximity or connected to the part on the productwhen the part is installed on the product, including determining therespective orientations of the respective parts, determining respectivedimensions or shapes of the respective products, and/or other respectivecharacteristics of the respective parts. For example, if the part is ascrew that is to be inserted into a threaded hole at the location on aportion the product, the IMC can identify or determine the dimensions(e.g., diameter and/or length) of the screw, the threading of the screw,the dimensions (e.g., diameter and/or length) of the threaded hole, thethreading of the threaded hole, and/or other respective characteristicsof the respective screw and threaded hole (and/or respectivecharacteristics of other parts of the product) based at least in part onsensor information from the sensor component and object recognition ofthe respective parts. If the IMC determines that dimensions andthreading of the screw correspond to the dimensions and threading of thethreaded hole and/or other assembly information confirms or corroboratesthat the screw is to be installed in that threaded hole, the IMC candetermine that the screw is to be installed in the threaded hole at thelocation on the portion of the product.

FIG. 15 illustrates a flow chart of an example method 1500 that candetermine hazards in an area, and can determine and present customizedinformation, comprising augmented reality information, via an ARD tofacilitate notifying or advising an ARD user of the hazards in the area,in accordance with various aspects and embodiments of the disclosedsubject matter. The area can be located in, for example, a building, aroom, a device, a cabinet, or another desired area. The method 1500 canbe employed by, for example, an IMC of an ARD.

At 1502, a location of a user associated with the ARD can be determinedbased at least in part on sensor information received from a sensorcomponent of the ARD. The IMC can receive sensor information relating tolocation of the ARD and associated user (e.g., user wearing the ARD)from one or more sensors (e.g., sensor(s) of a GPS or navigation system)of the sensor component. The IMC can determine the location of the ARDand the associated user based at least in part on the sensorinformation.

At 1504, a field of view of the user can be determined based at least inpart on the sensor information. For instance, a camera of the sensorcomponent can capture images of the field of view of the user wearing orotherwise associated with the ARD, and can provide sensor information(e.g., captured images) regarding the field of view of the user to theIMC. An orientation component, accelerometer, and/or gyroscope of thesensor component can sense orientation, position, and/or motion of theuser wearing or otherwise associated with the ARD, and can communicatethe sensor information relating to orientation, position, and/or motionof the user and ARD to the IMC. The IMC can determine the field of viewof the user associated with (e.g., wearing) the ARD based at least inpart on the sensor information received from the one or more sensors(e.g., camera, orientation component, accelerometer, and/or gyroscope, .. . ) of the sensor component.

At 1506, one or more hazards in an area associated with the user and ARDcan be identified based at least in part on the results of analyzing thesensor information received from the sensor component and/or informationrelating to the area that is obtained from one or more data sources. Thearea can span a region that is within a desired defined distance (e.g.,5 feet, 10 feet, 20 feet, or other desired larger or smaller distance)of the ARD and user. The area can include open and visible area portions(e.g., open and visible regions) and/or can include closed or obscuredarea portions (e.g., an inside area of a closed room, an inside area ofa closed cabinet, or an inside area of a device, . . . )

One or more sensors (e.g., heat sensor, electrical sensor, chemicalsensor, . . . ) of the sensor component can sense conditions (e.g.,environmental conditions), which can include one or more hazards, in thearea associated with the ARD and user. For example, the heat sensor ofthe ARD can detect temperature conditions of the area, includingtemperatures in the closed or obscured area portions (e.g., closed orobscured regions) of the area. The one or more sensors can communicatethe sensor information relating to the conditions to the IMC.

Additionally or alternatively, the IMC can obtain other informationrelating to conditions of the area from one or more data sources, suchas the data store or an external data source (e.g., via thecommunication network). For example, information relating to theconditions of the area can comprise information regarding one or morehazards (e.g., electrical hazards, temperature hazards, chemicalhazards, explosive hazards, . . . ) that are or potentially may be inthe area, including in the closed or obscured area portions.

The IMC can analyze the sensor information and/or the other information,and can determine or identify one or more hazards in the area, includingone or more hazards located in the closed or obscured area portions ofthe area, based at least in part on the results of such analysis.

At 1508, customized data, comprising augmented reality informationand/or virtual reality information, relating to the one or more hazardscan be determined. The IMC can determine and generate the customizedinformation relating to the one or more hazards based at least in parton the identification of the one or more hazards and results of theanalysis of the sensor information and/or the other information. Thecustomized information can comprise, textual information (e.g., words ornumbers describing the type of hazard and/or indicating a location of ahazard), audio information (e.g., audible sounds (e.g., words, numbers)describing the type of hazard and/or indicating a location of a hazard),visual information or indicators (e.g., a map, such as a heat map, thatcan indicate relative conditions, including hazards, of the area orfield of view and respective locations of such conditions and hazards;visual indicator that can indicate a hazard and its location), hapticinformation (e.g., the interface component can emit a vibration or othertype of haptic signal to the user when the ARD and user are in proximityto a hazard), and/or other information.

At 1510, the customized information relating to the one or more hazardsin the area or at least in the field of view of the user can bepresented to the user via the ARD (e.g., via the interface component ofthe ARD). The IMC can present or facilitate presenting, e.g., via theinterface component of the ARD, the customized information to the userto facilitate notifying or warning the user of the one or more hazardsin the area or at least in the field of view of the user. For example,in response to determining that a high temperature hazard is locatedinside a cabinet in the area, the IMC can facilitate presenting, via adisplay screen of the interface component, a visual indicator in aposition on the display screen that can correspond to the location ofthe cabinet in the area, wherein the visual indicator can indicate theinside of the cabinet contains a high temperature hazard. Additionallyor alternatively, the IMC can facilitate presenting a heat map on thedisplay screen, wherein the heat map can indicate relative temperatureconditions of respective regions (e.g., inside of the cabinet), objects(e.g., cabinet), etc., in the area or at least in the field of view ofthe user.

In order to provide a context for the various aspects of the disclosedsubject matter, FIGS. 16 and 17 as well as the following discussion areintended to provide a brief, general description of a suitableenvironment in which the various aspects of the disclosed subject mattercan be implemented.

With reference to FIG. 16, an example environment 1600 for implementingvarious aspects of the aforementioned subject matter includes a computer1612 (e.g., a computing component that can perform computingoperations). The computer 1612 includes a processing unit 1614, a systemmemory 1616, and a system bus 1618. The system bus 1618 couples systemcomponents including, but not limited to, the system memory 1616 to theprocessing unit 1614. The processing unit 1614 can be any of variousavailable processors. Multi-core microprocessors and othermultiprocessor architectures also can be employed as the processing unit1614.

The system bus 1618 can be any of several types of bus structure(s)including the memory bus or memory controller, a peripheral bus orexternal bus, and/or a local bus using any variety of available busarchitectures including, but not limited to, 8-bit bus, IndustrialStandard Architecture (ISA), Micro-Channel Architecture (MSA), ExtendedISA (EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB),Peripheral Component Interconnect (PCI), Universal Serial Bus (USB),Advanced Graphics Port (AGP), Personal Computer Memory CardInternational Association bus (PCMCIA), and Small Computer SystemsInterface (SCSI).

The system memory 1616 includes volatile memory 1620 and nonvolatilememory 1622. The basic input/output system (BIOS), containing the basicroutines to transfer information between elements within the computer1612, such as during start-up, is stored in nonvolatile memory 1622. Byway of illustration, and not limitation, nonvolatile memory 1622 caninclude read only memory (ROM), programmable ROM (PROM), electricallyprogrammable ROM (EPROM), electrically erasable PROM (EEPROM), or flashmemory. Volatile memory 1620 includes random access memory (RAM), whichacts as external cache memory. By way of illustration and notlimitation, RAM is available in many forms such as synchronous RAM(SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rateSDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), anddirect Rambus RAM (DRRAM).

Computer 1612 also includes removable/non-removable,volatile/nonvolatile computer storage media. FIG. 16 illustrates, forexample a disk storage 1624. Disk storage 1624 includes, but is notlimited to, devices like a magnetic disk drive, floppy disk drive, tapedrive, Jaz drive, Zip drive, LS-100 drive, flash memory card, or memorystick. In addition, disk storage 1624 can include storage mediaseparately or in combination with other storage media including, but notlimited to, an optical disk drive such as a compact disk ROM device(CD-ROM), CD recordable drive (CD-R Drive), CD rewritable drive (CD-RWDrive) or a digital versatile disk ROM drive (DVD-ROM). To facilitateconnection of the disk storage 1624 to the system bus 1618, a removableor non-removable interface is typically used such as interface 1626.

It is to be appreciated that FIG. 16 describes software that acts as anintermediary between users and the basic computer resources described insuitable operating environment 1600. Such software includes an operatingsystem 1628. Operating system 1628, which can be stored on disk storage1624, acts to control and allocate resources of the computer 1612.System applications 1630 take advantage of the management of resourcesby operating system 1628 through program modules 1632 and program data1634 stored either in system memory 1616 or on disk storage 1624. It isto be appreciated that one or more embodiments of the subject disclosurecan be implemented with various operating systems or combinations ofoperating systems.

A user enters commands or information into the computer 1612 throughinput device(s) 1636. Input devices 1636 include, but are not limitedto, a pointing device such as a mouse, trackball, stylus, touch pad,keyboard, microphone, joystick, game pad, satellite dish, scanner, TVtuner card, digital camera, digital video camera, web camera, and thelike. These and other input devices connect to the processing unit 1614through the system bus 1618 via interface port(s) 1638. Interfaceport(s) 1638 include, for example, a serial port, a parallel port, agame port, and a universal serial bus (USB). Output device(s) 1640 usesome of the same type of ports as input device(s) 1636. Thus, forexample, a USB port may be used to provide input to computer 1612, andto output information from computer 1612 to an output device 1640.Output adapters 1642 are provided to illustrate that there are someoutput devices 1640 like monitors, speakers, and printers, among otheroutput devices 1640, which require special adapters. The output adapters1642 include, by way of illustration and not limitation, video and soundcards that provide a means of connection between the output device 1640and the system bus 1618. It should be noted that other devices and/orsystems of devices provide both input and output capabilities such asremote computer(s) 1644.

Computer 1612 can operate in a networked environment using logicalconnections to one or more remote computers, such as remote computer(s)1644. The remote computer(s) 1644 can be a personal computer, a server,a router, a network PC, a workstation, a microprocessor based appliance,a peer device or other common network node and the like, and typicallyincludes many or all of the elements described relative to computer1612. For purposes of brevity, only a memory storage device 1646 isillustrated with remote computer(s) 1644. Remote computer(s) 1644 islogically connected to computer 1612 through a network interface 1648and then physically connected via communication connection 1650. Networkinterface 1648 encompasses communication networks such as local-areanetworks (LAN) and wide-area networks (WAN). LAN technologies includeFiber Distributed Data Interface (FDDI), Copper Distributed DataInterface (CDDI), Ethernet/IEEE 802.3, Token Ring/IEEE 802.5 and thelike. WAN technologies include, but are not limited to, point-to-pointlinks, circuit switching networks like Integrated Services DigitalNetworks (ISDN) and variations thereon, packet switching networks, andDigital Subscriber Lines (DSL).

Communication connection(s) 1650 refers to the hardware/softwareemployed to connect the network interface 1648 to the system bus 1618.While communication connection 1650 is shown for illustrative clarityinside computer 1612, it can also be external to computer 1612. Thehardware/software necessary for connection to the network interface 1648includes, for exemplary purposes only, internal and externaltechnologies such as, modems including regular telephone grade modems,cable modems and DSL modems, ISDN adapters, and Ethernet cards.

FIG. 17 is a schematic block diagram of a sample computing and/ornetworking environment 1700 with which the disclosed subject matter caninteract. The computing and/or networking environment 1700 can includeone or more clients 1702. The client(s) 1702 can be hardware and/orsoftware (e.g., threads, processes, computing devices). The computingand/or networking environment 1700 also can include one or more servers1704. The server(s) 1704 can also be hardware and/or software (e.g.,threads, processes, computing devices). The servers 1704 can housethreads to perform transformations by employing one or more embodimentsas described herein, for example. One possible communication between aclient 1702 and servers 1704 can be in the form of a data packet adaptedto be transmitted between two or more computer processes. The computingand/or networking environment 1700 can include a communication framework1706 that can be employed to facilitate communications between theclient(s) 1702 and the server(s) 1704. The client(s) 1702 are operablyconnected to one or more client data stores 1708 that can be employed tostore information local to the client(s) 1702. Similarly, the server(s)1704 are operably connected to one or more server data stores 1710 thatcan be employed to store information local to the servers 1704.

What has been described above includes examples of the disclosed subjectmatter. It is, of course, not possible to describe every conceivablecombination of components or methods for purposes of describing thedisclosed subject matter, but one of ordinary skill in the art mayrecognize that many further combinations and permutations of thedisclosed subject matter are possible. Accordingly, the disclosedsubject matter is intended to embrace all such alterations,modifications, and variations that fall within the spirit and scope ofthe appended claims.

In particular and in regard to the various functions performed by theabove described components, devices, circuits, systems and the like, theterms (including a reference to a “means”) used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (e.g., a functional equivalent), even though not structurallyequivalent to the disclosed structure, which performs the function inthe herein illustrated exemplary aspects of the disclosed subjectmatter. In this regard, it will also be recognized that the disclosedsubject matter includes a system as well as a computer-readable mediumhaving computer-executable instructions for performing the acts and/orevents of the various methods of the disclosed subject matter.

In addition, while a particular feature of the disclosed subject mattermay have been disclosed with respect to only one of severalimplementations, such feature may be combined with one or more otherfeatures of the other implementations as may be desired and advantageousfor any given or particular application. Furthermore, to the extent thatthe terms “includes,” and “including” and variants thereof are used ineither the detailed description or the claims, these terms are intendedto be inclusive in a manner similar to the term “comprising.”

It is to be appreciated and understood that components (e.g., ARD,sensor component, interface component, information management component,image capture component, audio sensor component, environment component,location component, scanner component, display component, audiocomponent, haptic component, key/control component, lens component,communicator component, operations manager component, recognitioncomponent, emphasis component, data determination component, processorcomponent, data store, or other components), as described with regard toa particular system or method, can include the same or similarfunctionality as respective components (e.g., respectively namedcomponents or similarly named components) as described with regard toother systems or methods disclosed herein.

In this application, the word “exemplary” is used to mean serving as anexample, instance, or illustration. Any aspect or design describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Rather, use of the wordexemplary is intended to present concepts in a concrete fashion.

Various aspects or features described herein may be implemented as amethod, apparatus, or article of manufacture using standard programmingand/or engineering techniques. The term “article of manufacture” as usedherein is intended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. For example, computerreadable media can include but are not limited to magnetic storagedevices (e.g., hard disk, floppy disk, magnetic strips . . . ), opticaldisks [e.g., compact disk (CD), digital versatile disk (DVD) . . . ],smart cards, and flash memory devices (e.g., card, stick, key drive . .. ).

As used in this application, the terms “component,” “system,”“platform,” “layer,” “controller,” “terminal,” “station,” “node,”“interface” are intended to refer to a computer-related entity or anentity related to, or that is part of, an operational apparatus with oneor more specific functionalities, wherein such entities can be eitherhardware, a combination of hardware and software, software, or softwarein execution. For example, a component can be, but is not limited tobeing, a process running on a processor, a processor, a hard disk drive,multiple storage drives (of optical or magnetic storage medium)including affixed (e.g., screwed or bolted) or removably affixedsolid-state storage drives; an object; an executable; a thread ofexecution; a computer-executable program, and/or a computer. By way ofillustration, both an application running on a server and the server canbe a component. One or more components can reside within a processand/or thread of execution, and a component can be localized on onecomputer and/or distributed between two or more computers. Also,components as described herein can execute from various computerreadable storage media having various data structures stored thereon.The components may communicate via local and/or remote processes such asin accordance with a signal having one or more data packets (e.g., datafrom one component interacting with another component in a local system,distributed system, and/or across a network such as the Internet withother systems via the signal). As another example, a component can be anapparatus with specific functionality provided by mechanical partsoperated by electric or electronic circuitry which is operated by asoftware or a firmware application executed by a processor, wherein theprocessor can be internal or external to the apparatus and executes atleast a part of the software or firmware application. As yet anotherexample, a component can be an apparatus that provides specificfunctionality through electronic components without mechanical parts,the electronic components can include a processor therein to executesoftware or firmware that provides at least in part the functionality ofthe electronic components. As further yet another example, interface(s)can include input/output (I/O) components as well as associatedprocessor, application, or application programming interface (API)components. While the foregoing examples are directed to aspects of acomponent, the exemplified aspects or features also apply to a system,platform, interface, layer, controller, terminal, and the like.

As used herein, the terms “to infer” and “inference” refer generally tothe process of reasoning about or inferring states of the system,environment, and/or user from a set of observations as captured viaevents and/or data. Inference can be employed to identify a specificcontext or action, or can generate a probability distribution overstates, for example. The inference can be probabilistic—that is, thecomputation of a probability distribution over states of interest basedon a consideration of data and events. Inference can also refer totechniques employed for composing higher-level events from a set ofevents and/or data. Such inference results in the construction of newevents or actions from a set of observed events and/or stored eventdata, whether or not the events are correlated in close temporalproximity, and whether the events and data come from one or severalevent and data sources.

In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B. In addition, the articles “a” and“an” as used in this application and the appended claims shouldgenerally be construed to mean “one or more” unless specified otherwiseor clear from the context to be directed to a singular form.

Furthermore, the term “set” as employed herein excludes the empty set;e.g., the set with no elements therein. Thus, a “set” in the subjectdisclosure includes one or more elements or entities. As anillustration, a set of controllers includes one or more controllers; aset of data resources includes one or more data resources; etc.Likewise, the term “group” as utilized herein refers to a collection ofone or more entities; e.g., a group of nodes refers to one or morenodes.

What is claimed is:
 1. A system, comprising: a memory that storesexecutable components; and a processor, operatively coupled to thememory, that executes the executable components, the executablecomponents comprising: an information management component configured toanalyze sensor data comprising historical sensor data relating to adevice, received from a set of sensors configured to detect conditionsassociated with at least one of a user or an environment associated withthe user, historical operational data relating to operation of thedevice, and lifetime rating data relating to a lifetime rating of thedevice, wherein the information management component is configured todetermine a context associated with the user and a condition relating tothe context, based on a result of the analysis, and determine customizedinformation based on the context of the user, wherein the conditionscomprise the condition that is associated with a maintenance taskinvolving the device, wherein the condition relates to a remainingportion of a lifetime usefulness of the device, and wherein thecustomized information comprises notification information relating tothe condition to notify the user about the condition relating to theremaining portion of the lifetime usefulness of the device; and aninterface component configured to present the customized information tothe user to facilitate performance of at least one task comprising themaintenance task.
 2. The system of claim 1, wherein the customizedinformation comprises at least one of augmented reality information orvirtual reality information.
 3. The system of claim 1, wherein, based onthe result of the analysis, the information management component isconfigured to determine a first risk level associated with a firstcomponent of the device and a second risk level associated with a secondcomponent of the device, display first risk level information relatingto the first risk level in a first region of a display screen of theinterface component that corresponds to a first viewing position of afirst view of the first component in a viewing lens associated with theuser, display second risk level information relating to the second risklevel in a second region of the display screen that corresponds to asecond viewing position of a second view of the second component in theviewing lens, and wherein the viewing lens is associated with thedisplay screen.
 4. The system of claim 1, wherein the sensor datarelates to infrared thermal measurements, voltage level measurements, orcurrent level measurements associated with the device, wherein thehistorical operational data indicates a length of time that the devicehas been in active service, a number of instances where operation of thedevice exceeded a rating value or a threshold value relating to theoperation the device, a number of times that the device has cycledbetween respective operational states, historical voltage levelmeasurements, historical current level measurements, or historicalinfrared thermal measurements, or wherein the lifetime rating datarelates to a first rating that indicates the lifetime usefulness of thedevice, a second rating that indicates the rating value or the thresholdvalue relating to the operation of the device, or a third rating of thedevice with regard to cycling of the device between the respectiveoperational states.
 5. The system of claim 1, wherein the informationmanagement component is configured to determine a wear level relating towear on the device or a stress level relating stress on the device,based on the result of the analysis of the sensor data, the historicaloperational data, and the lifetime rating data.
 6. The system of claim5, wherein the information management component is configured todetermine the remaining portion of the lifetime usefulness of the devicebased on the wear level or the stress level.
 7. The system of claim 1,wherein the information management component is configured to determinea set of risk levels associated with the device based on the result ofthe analysis of the sensor data, the historical operational data, andthe lifetime rating data, and wherein the set of risk levels comprise afirst risk level that indicates a first risk that the device fails, asecond risk level that indicates a second risk that failure of thedevice will result in failure of or damage to another device, a thirdrisk level that indicates a third risk relating to a cost associatedwith failure of the device, a fourth risk level that indicates a fourthrisk of there being a hazardous condition associated with failure of thedevice, or a fifth risk level that indicates a fifth risk relating to aloss of production associated with failure of the device.
 8. The systemof claim 1, wherein the condition is a first condition, wherein theconditions comprise a second condition, wherein the second condition isa hazardous condition, wherein the notification information is firstnotification information, wherein as part of the determination of thecontext associated with the user, based on the result of the analysis,the information management component is configured to detect thehazardous condition associated with performance of the maintenance taskby the user in connection with performing maintenance on the device, andwherein the customized information comprises second notificationinformation relating to the hazardous condition to notify the user aboutthe hazardous condition.
 9. The system of claim 1, wherein the device islocated inside of a cabinet of a piece of equipment, wherein theinformation management component is configured to determine devices,comprising the device, components associated with the devices, or one ormore hazardous conditions located inside the cabinet, based on theresult of the analysis, and determine the customized information basedon the devices, the components, or the one or more hazardous conditionsdetermined to be located inside the cabinet, and wherein the conditionscomprise the one or more hazardous conditions.
 10. The system of claim9, wherein, based on the sensor data or equipment-related data relatingto the piece of equipment, the information management component isconfigured to generate virtual objects that respectively represent thedevices, the components, or the one or more hazardous conditionsdetermined to be located inside the cabinet, and wherein the customizedinformation comprises the virtual objects.
 11. The system of claim 10,wherein the customized information, comprising the virtual objects,presented to the user via the interface component enable the user tovirtually visualize the devices, the components, or the one or morehazardous conditions located inside the cabinet before or withoutopening the cabinet.
 12. The system of claim 1, wherein the user is afirst user, wherein the interface component is a first interfacecomponent associated with the first user, wherein the informationmanagement component is configured to determine information, comprisingthe customized information, relating to the maintenance task, based onthe result of the analysis, and wherein the executable componentsfurther comprise: a second interface component configured to present theinformation, comprising the customized information, to a second user,wherein the information comprises visual information or audioinformation relating to the maintenance task, and wherein the visualinformation relates to a view of the device in connection withperformance of the maintenance task on the device by the first user. 13.The system of claim 1, wherein the device is a controller component,wherein, based on the sensor data or device-related data relating to thecontroller component, the information management component is configuredto generate virtual objects that respectively represent and areassociated with controls or switches of the controller component,wherein the virtual objects comprises a virtual object that representsand is associated with a control of the controls or a switch of theswitches, wherein the customized information comprises the virtualobjects, wherein the interface component is configured to receive inputdata from the user to manipulate and modify the virtual object, andwherein, in response to the input data, the information managementcomponent is configured to modify the virtual object to perform acorresponding modification to the control or the switch of thecontroller component.
 14. A method, comprising: analyzing, by a systemcomprising a processor, sensor data, comprising historical sensor datarelating to a device, historical operational data relating to operationof the device, and lifetime rating data relating to a lifetime rating ofthe device, wherein the sensor data is received from a set of sensorsconfigured to sense conditions associated with at least one of a user,the device, or an environment associated with the user; determining, bythe system, a context associated with the user and a condition relatingto the context, based on a result of the analyzing of the sensor data,the historical operational data, and the lifetime rating data, whereinthe conditions comprise the condition that is associated with amaintenance task involving the device, and wherein the conditionindicates a remaining portion of a lifetime usefulness of the device;determining, by the system, customized data based on the context of theuser and the condition, wherein the customized data comprisesnotification data relating to the condition to inform the user about thecondition that indicates the remaining portion of the lifetimeusefulness of the device; and presenting, by the system, the customizeddata to the user to facilitate performance of the maintenance task. 15.The method of claim 14, wherein the customized data comprises at leastone of augmented reality data or virtual reality data, wherein a portionof the customized data comprises visual data, wherein another portion ofthe customized data comprises audio data or haptic data, and wherein themethod further comprises: displaying, by the system, via a firstinterface, the visual data in a field of view of the user; andpresenting, by the system, via a second interface, the audio data or thehaptic data to the user.
 16. The method of claim 14, comprising: basedon the result of the analyzing of the sensor data, the historicaloperational data, and the lifetime rating data, determining, by thesystem, a first risk level associated with a first component of thedevice and a second risk level associated with a second component of thedevice; presenting, by the system, first risk level data relating to thefirst risk level in a first region of a display screen that correspondsto a first viewing position of a first view of the first component in aviewing lens associated with the user, wherein the viewing lens isassociated with the display screen; and presenting, by the system,second risk level data relating to the second risk level in a secondregion of the display screen that corresponds to a second viewingposition of a second view of the second component in the viewing lens.17. The method of claim 14, wherein the device is located inside of acabinet of a piece of equipment, and wherein the method furthercomprises: based on the sensor data or equipment-related data relatingto the piece of equipment, determining, by the system, devices,comprising the device, components associated with the devices, or one ormore hazardous conditions located inside the cabinet, wherein theconditions comprise the one or more hazardous conditions; generating, bythe system, virtual objects that respectively represent the devices, thecomponents, or the one or more hazardous conditions determined to belocated inside the cabinet, wherein the customized information comprisesthe virtual objects; and presenting, by the system, via an interface,the virtual objects, wherein the virtual objects are able to bevirtually interacted with by the user to enable the user to plan orrehearse performance of the maintenance task.
 18. The method of claim14, wherein the condition is a first condition, wherein the conditionscomprise a second condition that is a hazardous condition, wherein thenotification data is first notification data, and wherein the methodfurther comprises: based on the result of the analyzing, detecting, bythe system, the hazardous condition associated with performance of themaintenance task by the user in connection with performing maintenanceon the device, wherein the customized data comprises second notificationdata relating to the hazardous condition to inform the user about thehazardous condition.
 19. A non-transitory machine-readable medium havingstored thereon instructions that, in response to execution, cause asystem comprising a processor to perform operations, the operationscomprising: evaluating sensor information, comprising historical sensorinformation relating to a device, historical operational informationrelating to operation of the device, and lifetime rating informationrelating to a lifetime rating of the device, wherein the sensorinformation is received from a set of sensors that detect conditionsassociated with at least one of a user, the device, or an environmentassociated with the user; determining a context associated with the userand a condition relating to the context, based on a result of theanalyzing, wherein the conditions comprise the condition that isassociated with a maintenance job involving performing maintenance onthe device, and wherein the condition relates to a remaining portion ofa lifetime usefulness of the device; determining customized informationbased on the context of the user and the condition, wherein thecustomized information comprises notification information relating tothe condition to inform the user about the condition that relates to theremaining portion of the lifetime usefulness of the device; andcommunicating the customized information to the user to facilitateperformance of the maintenance job by the user.
 20. The non-transitorymachine-readable medium of claim 19, wherein the customized informationcomprises at least one of augmented reality information or virtualreality information, wherein a portion of the customized informationcomprises visual information, wherein another portion of the customizedinformation comprises audio information or haptic information, andwherein the operations further comprise: displaying, via a firstinterface component, the visual information in a field of view of theuser; and presenting, via a second interface component, the audioinformation or the haptic information to the user.